Handling Timeouts Gracefully with Python's urllib

Feb 6, 2024 ยท 2 min read

When fetching data from external websites and APIs, one common issue that arises is timeouts - where the server takes too long to respond and the request fails. Python's urllib module provides some useful tools to handle timeouts gracefully and retry requests when needed.

Setting Timeout Values

The urllib request objects allow setting a timeout value in seconds to control how long it will wait for a response before failing with a timeout error:

import urllib.request

req = urllib.request.Request('http://example.com')
req.timeout = 1.5 # seconds

Shorter timeouts are useful for failing fast, while longer values over 30 seconds may be needed for some slow APIs.

Catching Timeout Errors

When a timeout occurs, it will raise a URLError exception:

try:
    urllib.request.urlopen(req)
except urllib.error.URLError as e:
    print("Timeout error!", e)  

We can catch this and implement retry logic if needed.

Retrying with Exponential Backoff

A simple retry approach is an exponential backoff - where we retry after longer delays each time:

import time, math

tries = 0 
max_tries = 3
while tries < max_tries:
    try:
       urllib.request.urlopen(req) 
       break # request succeeded
    except urllib.error.URLError:
        tries += 1 
        wait_seconds = math.pow(2, tries) 
        print(f"Retry in {wait_seconds} seconds...")
        time.sleep(wait_seconds) 

This waits longer after each failure - helping balance resilience and not overloading a server.

Other Considerations

Other tips for handling timeouts:

  • Monitor for high timeout rates as an indicator of system issues
  • Use connection pooling when making many requests
  • Avoid retry loops on persistent failures
  • Getting timeout handling right goes a long way to building robust applications!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!