Stories from the Web Crawling trenches in CPU-bound

Why is multithreading not faster in python?

Author: Mohan Ganesan

Date: Mar 24, 2024

Python's multithreading capabilities are limited by the Global Interpreter Lock (GIL), but can still provide performance benefits for I/O-bound tasks. Tips include using multiprocessing for CPU-bound tasks and avoiding shared memory between threads.

Pushing Asyncio to the Limit: Understanding Concurrency Limits

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module in Python enables concurrent execution of code by running tasks asynchronously. It depends on factors like number of threads, nature of tasks, and settings.

When Async Python Outperforms Sync

Author: Mohan Ganesan

Date: Mar 17, 2024

Async programming in Python allows code to execute out of order while waiting on long-running tasks like network I/O. Async speeds up I/O-bound workloads but can be slower for heavy CPU processing. Always profile before and after to validate.

Multithreading in Python: Choosing the Right Model

Author: Mohan Ganesan

Date: Mar 17, 2024

Multithreading in Python can improve performance and responsiveness. Choose the right model based on use case and tradeoffs. Options include threading, multiprocessing, and asyncio.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!