Stories from the Web Crawling trenches in Python programs

Troubleshooting Python Request Timeouts

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python can sometimes result in timeouts due to slow network connection, overloaded API servers, short timeout values, or connection issues. To handle timeouts, you can check connectivity, increase the timeout duration, implement retries, handle exceptions, and assess for overload. Best practices to avoid timeouts include monitoring requests, stress testing remote APIs, implementing circuit breakers, and caching API response data.

What is the difference between asyncio and time sleep in Python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python provides asyncio module for concurrency and time.sleep for pausing execution. Use asyncio for parallelism and time.sleep carefully.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!