Stories from the Web Crawling trenches in multiprocessing

Combining AsyncIO and Multiprocessing in Python

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio library and multiprocessing module can be combined for improved resource utilization and cleaner code. Data passing between the two requires caution.

Why is Python Multithreading Slow and How to Speed It Up

Author: Mohan Ganesan

Date: Mar 17, 2024

Multithreading in Python seems slower due to the Global Interpreter Lock (GIL). Workarounds include multiprocessing for CPU-bound tasks and multithreading for I/O-bound tasks. External C/C++ libraries and newer Python versions also improve parallelism.

Does asyncio use multiple cores python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables concurrency within a single thread, but not parallelism across multiple threads or processes. However, by utilizing multiprocessing or multithreading, we can achieve true parallelism.

Faster Parallel Processing Alternatives to Multithreading in Python

Author: Mohan Ganesan

Date: Mar 17, 2024

Multithreading in Python allows concurrent execution of multiple threads within a process. However, it has limitations due to the GIL. Alternatives like multiprocessing, Numba, and Cython provide better parallelism and performance.

What is the difference between parallel and async in Python?

Author: Mohan Ganesan

Date: Mar 24, 2024

Python offers two options for performing multiple tasks simultaneously: parallel programming, which leverages multiple CPU cores, and asynchronous programming, which allows long-running functions to yield control back while waiting.

Why Python's Multithreading Perfoms Poorly (And What To Do About It)

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's multithreading capabilities are limited due to the GIL. Solutions like multiprocessing and asynchronous frameworks exist.

Is asyncio concurrent or parallel python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Asyncio provides concurrency, not parallelism. It shines for I/O bound work and can achieve high performance. Use multiprocessing for CPU intensive tasks.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!