Stories from the Web Crawling trenches in concurrent

Running multiple asyncio tasks

Author: Mohan Ganesan

Date: Mar 25, 2024

When writing async code in Python, asyncio provides two methods for running async tasks in parallel: asyncio.gather() and asyncio.create_task(). gather() bundles tasks and waits for them, while create_task() schedules background work.

Asyncio event loop

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module is a powerful tool for writing concurrent and asynchronous code. The event loop manages tasks and callbacks, allowing for efficient handling of thousands of concurrent requests.

Making HTTP Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library is a popular asynchronous HTTP client/server framework for Python. It allows you to make HTTP requests without blocking your application, perfect for building highly concurrent or asynchronous services.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!