Stories from the Web Crawling trenches in network requests

Making Fast Parallel Requests with Asyncio

Author: Mohan Ganesan

Date: Feb 3, 2024

Asyncio is a powerful Python library for performing asynchronous I/O operations and running multiple tasks concurrently. It allows creating asynchronous code that executes out of order while waiting on long-running operations like network requests.

Is Python async or sync?

Author: Mohan Ganesan

Date: Mar 24, 2024

New Python developers often get tripped up on the difference between asynchronous and synchronous execution. Asynchronous execution allows statements to run out of order without waiting. Python itself is synchronous, but it enables asynchronous execution through libraries like asyncio.

Does Python requests use urllib3?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python requests library provides a high-level interface for making HTTP requests, while urllib3 handles the low-level details.

When to use async python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python developers can use async code for faster and more efficient programming. Async is useful for network requests, file I/O, concurrency, and improving perceived performance. However, it should be avoided for CPU intensive tasks. Mixing async and sync code can cause deadlocks, and debugging async code can be challenging. Bridge between sync and async with asyncio.to_thread() and use purpose-built tools like aiomonitor for debugging.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!