Stories from the Web Crawling trenches in ClientSession

Making Concurrent Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

When building applications with aiohttp in Python, it's common to need to make multiple requests concurrently rather than sequentially. Use asyncio.gather, reuse session, and avoid limits with asyncio.Semaphore for better performance.

Secure HTTP Requests in Python with aiohttp ClientSession SSL

Author: Mohan Ganesan

Date: Feb 22, 2024

Making secure HTTPS requests in Python simplified with aiohttp ClientSession SSL functionality.

Making Async HTTP Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Python's requests library makes it easy to make synchronous HTTP requests in your code. But in async environments, like asyncio, you'll want to use an async HTTP client instead.

Properly Closing aiohttp Clients and Sessions

Author: Mohan Ganesan

Date: Mar 3, 2024

Properly close aiohttp ClientSession and connections to avoid resource leaks and TCP connection leaks over time.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!