Stories from the Web Crawling trenches in timeout settings

Troubleshooting Hanging Requests with Python Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies sending HTTP requests. Troubleshoot hanging requests by checking for network/connectivity issues, using timeout settings, implementing exponential backoff, and checking for deadlocks/race conditions.

Tuning aiohttp Request Timeouts for Optimal Performance

Author: Mohan Ganesan

Date: Mar 3, 2024

Managing request timeouts in aiohttp is crucial for good performance. Default timeouts may cause resource exhaustion and unresponsive UI. Tuning timeouts based on application load and setting them globally can prevent failures and improve user experience.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!