Stories from the Web Crawling trenches in threads

Making Asynchronous HTTP Requests in Python without Waiting for a Response

Author: Mohan Ganesan

Date: Feb 3, 2024

Make asynchronous HTTP requests in Python without blocking using the requests library, asyncio module, or threads/processes.

Making Concurrent Requests in Python: A Programmer's Guide

Author: Mohan Ganesan

Date: Nov 18, 2023

Handling multiple API calls and web scraping concurrently is critical for Python developers. This guide explores techniques for performant concurrent requests in Python.

How many threads does asyncio use python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asyncio is a powerful framework in Python that enables writing asynchronous, non-blocking code using a single-thread event loop. It allows concurrency through cooperative multitasking and the use of additional threads for CPU-bound work.

Python Threads vs Processes: Which is Faster and When to Use Each

Author: Mohan Ganesan

Date: Mar 24, 2024

When writing Python programs, developers often wonder if it's better to use threads or processes. Processes are generally faster and more robust, but have higher overhead. Threads require less resources to create, but come with their own challenges.

Does Python asyncio use threads?

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's asyncio module provides single-threaded concurrency using coroutines and an event loop. It can offload blocking IO and CPU-bound tasks to thread pools.

Async IO in Python: When and Why to Use It Over Threads

Author: Mohan Ganesan

Date: Mar 17, 2024

Leverage async I/O for non-CPU bound tasks that deal with network, disk, or user interactions for great performance gains. Stick to threads for intensive computational workloads.

Why coroutines are better than threads in python?

Author: Mohan Ganesan

Date: Mar 25, 2024

Coroutines in Python provide a lightweight alternative for concurrent programming without the overhead of threads. They are ideal for I/O bound workloads and enable simple, efficient, and scalable code.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!