Stories from the Web Crawling trenches in performance optimization

Making Asynchronous HTTP Requests in Python with aiohttp Connectors

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library provides a powerful tool for making asynchronous HTTP requests in Python. The aiohttp.TCPConnector manages connection pooling and reuse, allowing for improved performance and optimization of HTTP clients and services.

Is Beautiful soup slow?

Author: Mohan Ganesan

Date: Feb 5, 2024

Beautiful Soup is a popular Python library for web scraping. It can be optimized for faster scraping by using appropriate parsers, parsing only necessary data, caching, and performance profiling.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!