Smarter Retries with Python Requests

Feb 3, 2024 ยท 2 min read

Making HTTP requests is core to many Python applications. The popular requests library makes this easy, but sometimes requests fail unexpectedly. Instead of giving up after the first failure, it's smarter to retry failed requests to improve reliability.

Here are some tips for adding smart retries to Python requests:

Use Exponential Backoff

When retrying, use exponential backoff - gradually increase the wait time between retries. This reduces load on failing servers. Here's sample code:

from time import sleep

DELAY = 1 # seconds

for i in range(MAX_RETRIES):
    response = requests.get(url)

  except Exception as e:
    DELAY *= 2

This waits 1 second then 2, 4, 8 etc. Customize the backoff factor and max retries as needed.

Handle Specific Exceptions

Some exceptions indicate a retry won't help. Handle these separately:

from requests.exceptions import ConnectionError

  response = requests.get(url)
except ConnectionError:
  # Retry wouldn't help, re-raise
except Exception as e:
  # Retry other failures

This avoids wasted retries for known-bad failures like connection issues.

Use Retrying Packages

For more advanced retry logic, use a dedicated package like retrying. It handles backoffs, jitter delays and complex retry conditions.

Key Takeaways

  • Use exponential backoff between retries
  • Handle exceptions separately where retries won't help
  • For advanced logic, use a dedicated retrying package
  • Adding smart retries improves reliability at a low cost. Failures happen - with good retry handling your Python requests will bounce back smarter.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: