Stories from the Web Crawling trenches in parallel execution

Speed Up Your Asyncio Code with Thread Pools

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio is great for writing non-blocking network code in Python. But sometimes you have CPU-bound tasks that could benefit from parallel execution. That's where thread pools come in handy!

Asyncio gather usage

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module in Python provides powerful tools for writing asynchronous and concurrent code. One very useful function is asyncio.gather(), which allows you to simplify running multiple coroutines concurrently.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!