Stories from the Web Crawling trenches in pitfalls

When to use async python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python developers can use async code for faster and more efficient programming. Async is useful for network requests, file I/O, concurrency, and improving perceived performance. However, it should be avoided for CPU intensive tasks. Mixing async and sync code can cause deadlocks, and debugging async code can be challenging. Bridge between sync and async with asyncio.to_thread() and use purpose-built tools like aiomonitor for debugging.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!