Stories from the Web Crawling trenches in threading

Improving Performance of Python Requests with Threading

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests library provides a simple interface for making HTTP requests. Threading can help speed up requests by allowing multiple requests to be sent concurrently. Use thread pool, handle exceptions, watch for race conditions, use locks or queues for coordination. Threading improves performance for I/O-bound tasks. Beware of race conditions with shared data. Consider using grequests library for asynchronous requests.

What is the difference between parallel and async in Python?

Author: Mohan Ganesan

Date: Mar 24, 2024

Python offers two options for performing multiple tasks simultaneously: parallel programming, which leverages multiple CPU cores, and asynchronous programming, which allows long-running functions to yield control back while waiting.

Why is asyncio faster than threading python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module allows for non-blocking, asynchronous code execution, achieving better performance by minimizing blocking calls and maximizing CPU utilization.

Is asyncio python better than threading?

Author: Mohan Ganesan

Date: Mar 17, 2024

Async IO vs Threading in Python: A Practical Comparison. Async IO and threading are two options for concurrency in Python. This article compares their strengths and weaknesses, including performance, scalability, and library compatibility.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!