Stories from the Web Crawling trenches in exponential backoff

Troubleshooting Hanging Requests with Python Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

Python Requests library simplifies sending HTTP requests. Troubleshoot hanging requests by checking for network/connectivity issues, using timeout settings, implementing exponential backoff, and checking for deadlocks/race conditions.

Handling Errors Gracefully with Asyncio Retries

Author: Mohan Ganesan

Date: Mar 25, 2024

Implementing resilient retry logic in Asyncio apps using Python to handle transient errors and maintain availability.

Handling Timeouts Gracefully with Python's urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

When fetching data from external websites and APIs, handling timeouts gracefully and implementing retry logic with exponential backoff is crucial for building robust applications.

Handling Failed Requests in Python: Techniques for Resilience

Author: Mohan Ganesan

Date: Feb 3, 2024

Best practices for handling failed requests in Python: use try/except blocks, implement exponential backoff for retries, and use a circuit breaker pattern.

Smarter Retries with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Adding smart retries to Python requests improves reliability by using exponential backoff and handling exceptions separately.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!