How to fix ReadTimeout error in Python requests

Oct 22, 2023 ยท 6 min read

Overview of ReadTimeout Error

What is a ReadTimeout error?

A ReadTimeout error occurs when making requests using the Python requests module and indicates that the server failed to send any data in the allotted timeout period.

The requests module allows setting a timeout value in seconds, such as:

import requests

response = requests.get("<https://example.com>", timeout=10)

If the server takes longer than 10 seconds to respond, a ReadTimeout exception will be raised:

requests.exceptions.ReadTimeout: HTTPConnectionPool(host='example.com', port=80): Read timed out. (read timeout=10)

This timeout can occur due to a slow server, congested network, or if the server is explicitly blocking or rate limiting requests from your IP address.

When does it occur?

Some common scenarios that can trigger a ReadTimeout error include:

  • Making requests to an overloaded server that is slow to respond
  • Hitting rate limits on the number of requests allowed per minute/second
  • Making requests over an unreliable network connection
  • Server blocking requests without sending a response
  • Bug in application fails to write response in time
  • Firewall blocking access to server ports/resources
  • The timeout can occur intermittently, on certain endpoints, or for all requests if connectivity to the server is disrupted.

    Fixes for ReadTimeout Error

    Increase the Read Timeout Value

    An easy fix is to simply increase the timeout value passed to requests.get() to allow more time for the server to respond:

    response = requests.get("<https://example.com>", timeout=30)
    

    This may be necessary for exceptionally slow APIs or networks. However, simply increasing the timeout endlessly is not recommended.

    Use Try/Except to Handle the Error

    Wrapping the requests in a try/except block allows gracefully handling the ReadTimeout:

    try:
      response = requests.get("<https://example.com>", timeout=10)
    except requests.ReadTimeout:
      print("Timeout occurred, retrying...")
      response = requests.get("<https://example.com>", timeout=30)
    

    This prevents the request from halting on a timeout and allows retrying with a longer timeout.

    Enable Retries in Requests

    The requests module can automatically retry failed requests using the Retry object:

    from requests.adapters import HTTPAdapter
    from requests.packages.urllib3.util.retry import Retry
    
    retry_strategy = Retry(
      total=3,
      backoff_factor=1,
      status_forcelist=[429, 500, 502, 503, 504],
      method_whitelist=["HEAD", "GET", "OPTIONS"]
    )
    
    adapter = HTTPAdapter(max_retries=retry_strategy)
    http = requests.Session()
    http.mount("https://", adapter)
    

    This will retry on timeout and 5xx errors, preventing false positives.

    Check for Server Blocking

    If you are encountering frequent ReadTimeouts, the server may be blocking or rate limiting requests from your IP address or user agent.

    Try sending requests through a proxy or rotating IPs to determine if you are being blocked. Scraping-specific services like ScrapingBee can also circumvent blocks.

    Use Asynchronous Requests

    For Python 3.7+, the asyncio module allows making asynchronous requests concurrently without blocking:

    import asyncio
    import aiohttp
    
    async def fetch(url):
      async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
          return response
    
    async def main():
      await asyncio.gather(
        fetch("<https://example.com/1>"),
        fetch("<https://example.com/2>")
      )
    
    asyncio.run(main())
    

    This prevents a slow response from one request blocking others.

    Confirm Server is Responding

    It's also worth directly testing the endpoint you are requesting through cURL or Postman to confirm the server is up and responding properly.

    A ReadTimeout likely indicates a server-side issue or network disruption between your application and the server. Eliminating these possibilities first simplifies troubleshooting.

    Additional Tips

  • Set a connect timeout as well to fail faster if the connection fails
  • Avoid setting timeout too high, better to retry shorter requests
  • Use exponential backoff when retrying requests
  • Log and monitor timeouts to identify problematic endpoints
  • Check status of target server through Ping, Telnet
  • Inspect headers for rate limiting info
  • Try from different networks/IPs if possible
  • Set proxy and user agent to avoid blocks
  • In summary, ReadTimeout errors can be addressed by optimizing timeouts, implementing robust retry logic, confirming the server is responsive, and using tools to avoid blocks. The ideal fix depends on the specific request, endpoint, and infrastructure.

    FAQ

    How do you handle request timeout in Python?

    You can set timeouts on requests in Python using the timeout parameter. This specifies the number of seconds to wait for a response before timing out:

    import requests
    
    response = requests.get('<http://example.com>', timeout=3)
    

    What is the Python request exception for ConnectTimeout?

    The requests.exceptions.ConnectTimeout exception is raised if the request times out while trying to connect to the remote server. This indicates the connection could not be established within the timeout period.

    How do I increase read timeout in Python?

    Set the timeout parameter to a higher value (in seconds). You can also set read_timeout separately for just the read operation.

    response = requests.get('<http://example.com>', timeout=10, read_timeout=15)
    

    What happens on timeout in requests Python?

    The requests.exceptions.Timeout exception is raised if the request exceeds the timeout without receiving a response. This could occur during connection, read/send operations.

    What is the cause of read timeout?

    A read timeout occurs if a response is not received within the read_timeout period once the connection is established. This may indicate network issues or an overloaded server.

    What is the default timeout for a Python request?

    The default timeout for requests is None, meaning no timeout. The connection can remain open indefinitely.

    How to check timeout error in Python?

    Check if a requests.exceptions.Timeout exception was raised when making the request. The error message will indicate if it was a connect or read timeout.

    What is a timeout error Python?

    The requests.exceptions.Timeout exception in Python indicates a request has exceeded its timeout without receiving a response from the server.

    Does Python requests have a timeout?

    Yes, you can set timeouts on requests in Python using the timeout and read_timeout parameters. By default no timeouts are set.

    Why is my request timed out?

    Possible reasons your request is timing out include: slow network, overloaded server, incorrect timeout value set, or the server is just taking too long to process your request. Check your network connectivity and server health.

    What are the different types of request timeouts?

    The main timeouts in requests are connect timeout and read timeout. Connect timeout occurs if no connection can be made within the timeout period. Read timeout occurs during reading the response, after connection is established.

    How do you handle timeout?

    To handle timeout, catch the requests.exceptions.Timeout exception and implement a retry strategy and/or display an appropriate error message to the user. You may also need to investigate and fix the root cause (network issue, slow server, etc).

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!