Stories from the Web Crawling trenches in Queue Size Limiting

Implementing Scalable Async I/O with Python Asyncio Queues

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio queues provide a great way to pass data between asynchronous tasks in Python. They enable building scalable asynchronous I/O flows without some of the downsides of threads or processes.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!