Stories from the Web Crawling trenches in asynchronous code

Combining AsyncIO and Multiprocessing in Python

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio library and multiprocessing module can be combined for improved resource utilization and cleaner code. Data passing between the two requires caution.

Making the Most of asyncio: Adding Tasks to Event Loops

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module in Python provides infrastructure for writing asynchronous code using the async/await syntax. The event loop is at the heart of asyncio and manages task execution. Enqueue tasks with loop.create_task() or ensure_future().

Testing Asynchronous Code with Aiohttp Test Utilities

Author: Mohan Ganesan

Date: Mar 3, 2024

The aiohttp library in Python provides utilities for testing asynchronous code. Use aiohttp.test_utils module to test web APIs and apps.

Requests vs urllib vs httpx vs aiohttp

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python: comparing Requests, urllib, httpx, and aiohttp. Requests is the easiest, urllib is lower-level, httpx adds advanced features, and aiohttp is for asyncio-based code.

Understanding Asyncio Event Loops in Python

Author: Mohan Ganesan

Date: Mar 25, 2024

The event loop is the core of asyncio in Python, handling asynchronous code and callbacks. Properly managing the event loop is key to writing efficient asyncio programs.

Asyncio event loop

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module is a powerful tool for writing concurrent and asynchronous code. The event loop manages tasks and callbacks, allowing for efficient handling of thousands of concurrent requests.

Why is asyncio faster than threading python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module allows for non-blocking, asynchronous code execution, achieving better performance by minimizing blocking calls and maximizing CPU utilization.

Is asyncio deprecated python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asyncio is an integral part of Python, providing efficient framework for writing asynchronous code. It allows concurrent execution without the complexity of threads or multiprocessing.

Is asyncio a standard library python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Async IO is a useful concurrent programming framework in Python's standard library for executing multiple tasks concurrently within a single thread.

What is the difference between async and synchronous await?

Author: Mohan Ganesan

Date: Mar 24, 2024

JavaScript's asynchronous nature can be managed using callback functions or the async/await syntax. Callbacks can lead to 'callback hell', while async/await allows for synchronous-looking code that remains asynchronous. Mixing async and synchronous code can be tricky, and understanding when code yields execution takes practice.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!