Stories from the Web Crawling trenches in concurrency

Speeding up Python Requests using gzip and other techniques

Author: Mohan Ganesan

Date: Dec 6, 2023

Making Concurrent Requests in Python: A Programmer's Guide

Author: Mohan Ganesan

Date: Nov 18, 2023

Handling multiple API calls and web scraping concurrently is critical for Python developers. This guide explores techniques for performant concurrent requests in Python.

Unlocking Async Performance with Asyncio Redis

Author: Mohan Ganesan

Date: Mar 25, 2024

Redis is a popular in-memory data store known for its speed and versatility. By combining Redis with Python's asyncio module, you can build extremely fast and scalable applications.

Asyncio gathering task results

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio.gather() function is useful for launching multiple coroutines concurrently and waiting for their results. It is commonly used for coordinating web requests, IO work, and parallel flows.

How to Build a Super Simple HTTP Proxy in Scala in Just 20 Lines of Code

Author: Mohan Ganesan

Date: Oct 1, 2023

Scala makes it easy to build networked applications with concise syntax and strong libraries. Here is an HTTP proxy server in Scala using Akka in just 20 lines of code. It is prone to get blocked due to single IP usage, but a rotating proxy service like Proxies API can solve IP blocking problems instantly.

Making the Most of asyncio: Adding Tasks to Event Loops

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module in Python provides infrastructure for writing asynchronous code using the async/await syntax. The event loop is at the heart of asyncio and manages task execution. Enqueue tasks with loop.create_task() or ensure_future().

Improving Performance of Python Requests with Threading

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests library provides a simple interface for making HTTP requests. Threading can help speed up requests by allowing multiple requests to be sent concurrently. Use thread pool, handle exceptions, watch for race conditions, use locks or queues for coordination. Threading improves performance for I/O-bound tasks. Beware of race conditions with shared data. Consider using grequests library for asynchronous requests.

Understanding Asyncio Coroutines and Tasks in Python

Author: Mohan Ganesan

Date: Mar 17, 2024

Asynchronous programming in Python using coroutines and tasks. Coroutines define asynchronous behavior, while tasks actually run the coroutines and enable concurrency.

Boosting Your Discord Bot's Performance with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

Build high-performance Discord bots with aiohttp, the leading asynchronous HTTP client for Python, to prevent blocking and improve concurrency.

Does asyncio use multiple cores python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables concurrency within a single thread, but not parallelism across multiple threads or processes. However, by utilizing multiprocessing or multithreading, we can achieve true parallelism.

Concurrency in Python: Understanding Asyncio vs Synchronous Code

Author: Mohan Ganesan

Date: Mar 17, 2024

Python is often used for building complex applications that handle multiple tasks concurrently. Understanding the difference between asyncio and synchronous code is key to writing efficient, scalable Python programs.

What is the difference between asyncio and time sleep in Python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python provides asyncio module for concurrency and time.sleep for pausing execution. Use asyncio for parallelism and time.sleep carefully.

Is Python asynchronous or synchronous?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables asynchronous I/O for improved concurrency. Use asyncio for I/O-bound tasks and when concurrency is needed.

Making Asynchronous HTTP Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library provides simple interface for making HTTP requests. Supports synchronous and asynchronous requests using threads or processes.

Concurrency and Thread Safety in Python's asyncio

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables concurrency within a single thread using an event loop. Sharing data between coroutines is thread-safe. Multithreading requires new event loops and explicit synchronization. Blocking code must execute in threads to avoid blocking the event loop. Following these best practices ensures efficient, thread-safe asyncio code.

What is the difference between asyncio and multithreading python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python developers often need to make their programs concurrent to improve performance. The two main options for concurrency in Python are asyncio and multithreading.

How many threads does asyncio use python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asyncio is a powerful framework in Python that enables writing asynchronous, non-blocking code using a single-thread event loop. It allows concurrency through cooperative multitasking and the use of additional threads for CPU-bound work.

Async IO and Futures in Python: What's the Difference?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asynchronous programming in Python with asyncio and futures. asyncio provides infrastructure for async I/O concurrency while futures represent eventual results of asynchronous operations.

Does asyncio run in parallel python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables concurrency, not parallelism, by using coroutines and an event loop.

Is asyncio python better than threading?

Author: Mohan Ganesan

Date: Mar 17, 2024

Async IO vs Threading in Python: A Practical Comparison. Async IO and threading are two options for concurrency in Python. This article compares their strengths and weaknesses, including performance, scalability, and library compatibility.

Does asyncio use multiple cores?

Author: Mohan Ganesan

Date: Mar 24, 2024

Asyncio enables concurrency, but not parallelism by default. You can achieve parallelism by integrating thread pools and process pools.

Asyncio Concurrency in Python: Unlocking Asynchronous Magic

Author: Mohan Ganesan

Date: Mar 25, 2024

Concurrency is essential for building responsive and scalable applications. Asyncio in Python allows for asynchronous code, making the most of hardware resources.

Python Threads vs Processes: Which is Faster and When to Use Each

Author: Mohan Ganesan

Date: Mar 24, 2024

When writing Python programs, developers often wonder if it's better to use threads or processes. Processes are generally faster and more robust, but have higher overhead. Threads require less resources to create, but come with their own challenges.

Is asyncio part of Python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables non-blocking concurrency, improving performance, scalability, and user experience.

Why Python's Multithreading Perfoms Poorly (And What To Do About It)

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's multithreading capabilities are limited due to the GIL. Solutions like multiprocessing and asynchronous frameworks exist.

What is the difference between asyncio and queue?

Author: Mohan Ganesan

Date: Mar 24, 2024

Asynchronous programming in Python with asyncio and queues. asyncio for I/O bound tasks, queues for CPU bound work. Different concurrency models and performance tradeoffs.

Pushing Asyncio to the Limit: Understanding Concurrency Limits

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module in Python enables concurrent execution of code by running tasks asynchronously. It depends on factors like number of threads, nature of tasks, and settings.

Does asyncio run in single thread python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module allows concurrent code using a single-threaded event loop model, providing performance benefits for I/O bound workloads.

Does Python asyncio use threads?

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's asyncio module provides single-threaded concurrency using coroutines and an event loop. It can offload blocking IO and CPU-bound tasks to thread pools.

Async IO and Generators: Key Differences in Python

Author: Mohan Ganesan

Date: Mar 24, 2024

Async IO and generators are powerful asynchronous programming concepts in Python with key differences. Generators produce data on demand, while Async IO enables concurrent work. Both are useful for different scenarios and can be used together to write highly scalable programs.

When to use async python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python developers can use async code for faster and more efficient programming. Async is useful for network requests, file I/O, concurrency, and improving perceived performance. However, it should be avoided for CPU intensive tasks. Mixing async and sync code can cause deadlocks, and debugging async code can be challenging. Bridge between sync and async with asyncio.to_thread() and use purpose-built tools like aiomonitor for debugging.

Why coroutines are better than threads in python?

Author: Mohan Ganesan

Date: Mar 25, 2024

Coroutines in Python provide a lightweight alternative for concurrent programming without the overhead of threads. They are ideal for I/O bound workloads and enable simple, efficient, and scalable code.

What is a socket in Python?

Author: Mohan Ganesan

Date: Feb 20, 2024

Sockets are a key concept in network programming that allow communication between processes or applications. In Python, sockets are enabled through the socket library. Client sockets are used to initiate communication with a server, while server sockets listen for incoming connections. Sockets enable bidirectional communication through sending and receiving data, and can handle multiple client connections concurrently.

When Async Python Outperforms Sync

Author: Mohan Ganesan

Date: Mar 17, 2024

Async programming in Python allows code to execute out of order while waiting on long-running tasks like network I/O. Async speeds up I/O-bound workloads but can be slower for heavy CPU processing. Always profile before and after to validate.

What are the advantages of asyncio in Python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module opens up a whole new world of asynchronous programming, allowing code to execute concurrently and resulting in huge performance gains for I/O-bound applications.

Multithreading in Python: Choosing the Right Model

Author: Mohan Ganesan

Date: Mar 17, 2024

Multithreading in Python can improve performance and responsiveness. Choose the right model based on use case and tradeoffs. Options include threading, multiprocessing, and asyncio.

Is asyncio concurrent or parallel python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asyncio provides concurrency, not parallelism. It shines for I/O bound work and can achieve high performance. Use multiprocessing for CPU intensive tasks.

Concurrency in Python: Understanding Asyncio and Futures

Author: Mohan Ganesan

Date: Mar 24, 2024

Python provides powerful tools for handling concurrency and parallelism with asyncio and futures. Asyncio enables asynchronous I/O handling in a single thread, while futures handle parallelism across threads/processes.

Achieving Speed with Asyncio in Python

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's asyncio library enables concurrency for improved performance, but not parallelism. It allows efficient use of I/O resources within a single thread.

Simplifying Asynchronous Code in Python with async and await

Author: Mohan Ganesan

Date: Mar 17, 2024

Async programming in Python using async/await simplifies writing non-blocking code that runs concurrently, making it ideal for high throughput and scalability in network apps.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!