Handling Failed Requests in Python: Techniques for Resilience

Feb 3, 2024 ยท 2 min read

When working with external services and APIs in Python, dealing with failed requests is inevitable. However, with some thoughtful error handling and retry logic, you can make your Python application much more resilient. Here are some best practices for handling failed requests:

Use Try/Except Blocks to Catch Errors

Wrapping API calls or network requests in try/except blocks allows you to catch errors cleanly and route execution based on different exception types:

    response = requests.get('http://api.example.com/data')
except requests.exceptions.Timeout:
    # Timeout occurred
except requests.exceptions.HTTPError:
    # Unsuccessful status code returned

This makes error handling explicit instead of failing silently.

Exponential Backoff for Retries

When retrying failed requests, use an exponential backoff algorithm that gradually increases the wait time between retries. This avoids bombarding unstable APIs. A simple implementation with the time module:

retry_count = 0
retry_delay = 1 # seconds

while True:
        response = requests.get(url) 
    except Exception:
        if retry_count < 5: 
            retry_delay *= 2
            retry_count += 1

Circuit Breaker Pattern

Once failures reach a threshold, use a "circuit breaker" to stop making requests for a period of time to allow the API to recover before trying again. This prevents wasting resources on recurring failed calls.

By planning for and handling failures systematically, you can build Python programs that gracefully handle unreliable APIs and network issues. The key is having clear error handling flow and not retrying endlessly.

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: