How to Handle Timeout error in Python requests

Oct 22, 2023 ยท 7 min read

Introduction

Timeouts are a critical aspect of making requests in Python. When sending requests to external servers or APIs, you don't want your program to hang indefinitely waiting for a response. This is where timeouts come in - they allow you to set a maximum time limit for requests to complete before raising an exception.

What are timeouts and why are they important?

A timeout is a predefined time limit for a request to a server to complete. In the Python requests library, we set timeouts to prevent:

  • Requests waiting indefinitely for a response that may never arrive from a slow/problematic server. This can freeze up the application.
  • Wastage of resources like CPU cycles, memory, threads that are blocked waiting on requests.
  • Building resilience by cutting off problematic requests and allowing retry logic.
  • Hence it is a good practice to always use appropriate timeouts when sending requests. The requests library doesn't apply any default timeouts, so you have to explicitly configure them.

    Setting Timeouts in Requests

    The requests library provides flexible ways to configure timeouts both at a per-request level and globally.

    Setting a timeout for the entire request

    You can set a timeout for any request using the timeout parameter. Pass the number of seconds to wait as a float:

    import requests
    
    response = requests.get('<https://api.example.com/data>', timeout=2.5)
    

    This will raise a Timeout exception if the request does not complete within 2.5 seconds.

    Timeout parameter

    The timeout can be set on any request:

    requests.get(url, timeout=3)
    requests.post(url, timeout=4.5)
    

    Apply it to any long-running requests that could hang.

    Example

    import requests
    
    try:
      response = requests.get('<https://slow-api.com/data>', timeout=1)
    except requests.Timeout:
      print("Request timed out!")
    

    This guarantees the GET request will fail if it takes over 1 second.

    Setting connect and read timeouts separately

    For more control, you can specify separate connect and read timeouts using a tuple:

    requests.get(url, timeout=(3, 10)) # 3s connect, 10s read
    

    This is useful when you want to control the timeout behavior accurately.

    Specifying a tuple

    The first value sets the connection timeout in seconds. This is the time to establish the connection to the server.

    The second value sets the read timeout i.e. the time to wait between consecutive read operations.

    Example

    import requests
    
    try:
      requests.get('<https://api.example.com>', timeout=(2, 5))
    except requests.Timeout:
      # Handle timeout
    

    This will timeout if connection takes over 2 seconds or if the server is slow sending data and stalls for >5 seconds during the request.

    No timeout by default

    Note that if you don't specify a timeout, requests will not timeout at all for that request. It will keep waiting for a response indefinitely.

    response = requests.get('<https://api.example.com>') # no timeout!
    

    So always set appropriate timeouts to avoid hanging requests.

    Timeout values

    Units (seconds vs milliseconds)

    The timeout should be specified in seconds as a float or integer. Other libraries like JavaScript use milliseconds so watch out for this difference.

    Recommended values

    Typical timeout values range from 1 second to 60 seconds for most requests. However, choose based on expected server response times. For fast internal API calls, a timeout of 0.1 - 0.5 seconds is also common.

    Handling Timeouts

    When a timeout occurs, the requests library raises timeout-related exceptions that you need to handle gracefully:

    Catching timeout exceptions

    Use try-except blocks to catch timeouts and handle the failure scenarios:

    ConnectTimeout

    Raised if a connection couldn't be established within the connect timeout.

    try:
      requests.get(url, timeout=1)
    except requests.ConnectTimeout:
      # Couldn't connect in time
    

    ReadTimeout

    Timeout occurred because the server didn't send any data for the read timeout period.

    try:
       requests.get(url, timeout=(2, 1))
    except requests.ReadTimeout:
       # Server sent no data
    

    RequestException

    Catch the base RequestException to handle all request-related errors including timeouts.

    Retrying on timeout

    For temporary timeouts, you can use exponential backoff to retry the request 2-3 times before throwing the exception. The urllib3 retry module implements this.

    Timeout behavior for streaming requests

    For streaming requests, the read timeout applies per chunk not the total response time. So streams won't be cut off abruptly even if the total time exceeds the timeout.

    Using third party libraries like eventlet

    If you need to enforce a total time limit for the request completion, you can use eventlet:

    import eventlet
    with eventlet.Timeout(10):
      requests.get(url)
    

    This will forcibly raise an exception if the request exceeds 10 seconds.

    Advanced Timeout Configuration

    You can configure timeouts in additional ways:

    Timeout config at session level

    Set timeout globally for a session by setting session.timeout which applies to all requests from that session:

    s = requests.Session()
    s.timeout = 10 # 10 seconds for all requests in s
    

    Custom retry logic for timeouts

    Use the Retry class to implement backoffs, custom retry criteria etc. for handling transient timeouts:

    from requests.adapters import HTTPAdapter
    from urllib3.util import Retry
    
    retry_strategy = Retry(
      total=3,
      backoff_factor=1,
      status_forcelist=[500, 502, 503],
      method_whitelist=["HEAD", "GET", "OPTIONS"]
    )
    
    adapter = HTTPAdapter(max_retries=retry_strategy)
    http = requests.Session()
    http.mount("https://", adapter)
    

    Global timeout configuration

    Set environment variables like REQUESTS_TIMEOUT to set global timeout duration that apply unless explicitly overridden.

    Other Considerations

    Effect on performance

    Lower timeouts reduce request time but may cause more errors. Higher values improve reliability but have slower performance. tuning may be needed based on your requirements.

    Tradeoffs between timeouts and errors

    Overly short timeouts lead to more timeout errors being raised, needing catch and retry logic. Too long timeouts won't provide the benefits of cutting off slow responses.

    Alternatives to requests for better timeout handling

    If you need very customizable timeout behavior, consider using a non-blocking I/O HTTP client like httpx or asyncio.

    Conclusion

    Timeouts are a critical configuration required when making requests to unreliable servers and APIs. The requests library provides flexible options to set timeouts at both global and per-request levels. Make sure to catch and handle timeout exceptions like ConnectTimeout and ReadTimeout appropriately in your code. Carefully tune timeout values and retry logic based on your specific requirements for performance vs reliability.

    Here is an FAQ on handling request timeouts in Python:

    How do I set a timeout for requests in Python?

    You can set a timeout by passing the timeout parameter while making the request. Set it to the number of seconds to wait before timing out:

    requests.get(url, timeout=5)
    

    What's the default timeout in the Python requests library?

    By default, there is no timeout applied in the requests library. Requests will wait indefinitely if no timeout is specified.

    How do I handle timeout errors in Python requests?

    Use try-except blocks to catch Timeout, ConnectTimeout or ReadTimeout exceptions. Then implement retry logic or display an error message to the user.

    What's the difference between connect vs read timeout?

    Connect timeout is the time to establish the connection to the server. Read timeout is the time between consecutive read operations while receiving the response. Specify them separately like:

    requests.get(url, timeout=(2, 5)) # 2s connect, 5s read
    

    How can I set timeouts globally for all requests?

    Set the timeout attribute on a requests Session which applies to all requests made from that session:

    s = requests.Session()
    s.timeout = 10
    

    What's a good timeout value to start with?

    Try a value between 1-5 seconds for most API requests. For faster internal APIs, 0.1 - 0.5 seconds is also reasonable. Adjust based on your specific API response times.

    How can I retry requests on timeout?

    Use urllib3's Retry class to implement backoffs and retry timeouts. Also look at the requests.Session.resolve_redirects parameter.

    How do I increase timeouts for slow responding APIs?

    Pass a higher timeout value in seconds while calling the API. Also consider retry logic and checking for server-side fixes.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!