Stories from the Web Crawling trenches in run_in_executor

Making Asynchronous Code Synchronous in aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library in Python allows for asynchronous HTTP requests. This article covers techniques to integrate aiohttp with synchronous code or external libraries, including using run_in_executor(), asyncio.to_thread(), running an event loop in a thread, and the nest_asyncio decorator.

Speed Up Your Asyncio Code with Thread Pools

Author: Mohan Ganesan

Date: Mar 25, 2024

Asyncio is great for writing non-blocking network code in Python. But sometimes you have CPU-bound tasks that could benefit from parallel execution. That's where thread pools come in handy!

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!