Stories from the Web Crawling trenches in async tasks

Running multiple asyncio tasks

Author: Mohan Ganesan

Date: Mar 25, 2024

When writing async code in Python, asyncio provides two methods for running async tasks in parallel: asyncio.gather() and asyncio.create_task(). gather() bundles tasks and waits for them, while create_task() schedules background work.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!