Stories from the Web Crawling trenches in run_in_executor

Speed Up Your Asyncio Code with Thread Pools

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio is great for writing non-blocking network code in Python. But sometimes you have CPU-bound tasks that could benefit from parallel execution. That's where thread pools come in handy!

Making Asynchronous Code Synchronous in aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library in Python allows for asynchronous HTTP requests. This article covers techniques to integrate aiohttp with synchronous code or external libraries, including using run_in_executor(), asyncio.to_thread(), running an event loop in a thread, and the nest_asyncio decorator.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!