Stories from the Web Crawling trenches in race conditions

Troubleshooting Hanging Requests with Python Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies sending HTTP requests. Troubleshoot hanging requests by checking for network/connectivity issues, using timeout settings, implementing exponential backoff, and checking for deadlocks/race conditions.

Improving Performance of Python Requests with Threading

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests library provides a simple interface for making HTTP requests. Threading can help speed up requests by allowing multiple requests to be sent concurrently. Use thread pool, handle exceptions, watch for race conditions, use locks or queues for coordination. Threading improves performance for I/O-bound tasks. Beware of race conditions with shared data. Consider using grequests library for asynchronous requests.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!