Stories from the Web Crawling trenches in Semaphore

Making Concurrent Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

When building applications with aiohttp in Python, it's common to need to make multiple requests concurrently rather than sequentially. Use asyncio.gather, reuse session, and avoid limits with asyncio.Semaphore for better performance.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!