Enable Detailed HTTP Debug Logging in Python Requests

Feb 3, 2024 ยท 2 min read

When working with the popular Python Requests library for making HTTP requests, it can be useful to enable more verbose debug logging to see exactly what is happening under the hood. By default, Requests doesn't log much, but with a few tweaks you can get insightful debug output.

Why Enable Debug Logging

There are a few great reasons to turn on debug logging in Requests:

  • See the full HTTP request and response including headers and body. This is helpful for debugging issues with APIs.
  • Inspect redirects, cookies, and authentication flow.
  • Double check that compression, proxies, and other settings are behaving as expected.
  • Learn more about the Requests internals if you ever need to debug tricky problems.
  • So if you need more visibility into your Python HTTP requests, debug logging is there for you.

    Quick Example

    Enabling debug logging in Requests just takes a few lines of code:

    import logging
    import requests
    
    # These two lines enable debugging at httplib level (requests->urllib3->http.client)
    # You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
    # The only thing missing will be the response.body which is not logged.
    try:
        import http.client as http_client
    except ImportError:
        # Python 2
        import httplib as http_client
    http_client.HTTPConnection.debuglevel = 1
    
    logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
    logging.getLogger().setLevel(logging.DEBUG)
    requests_log = logging.getLogger("requests.packages.urllib3")
    requests_log.setLevel(logging.DEBUG)
    requests_log.propagate = True
    
    requests.get('http://httpbin.org/headers')

    This uses the logging module to tap into the debug messages of both Requests and the underlying urllib3 HTTP client.

    The key things it does:

  • Enables debuglevel in http.client to log HTTP headers
  • Sets base logging to DEBUG level
  • Specifies debug logging for Requests and urllib3
  • Allows propagation from Requests to root logger
  • Now you'll see verbose logs on the console for the full request and response flow!

    Conclusion

    Debug logging in Requests provides much deeper insight and can save hours of headache debugging HTTP issues. Give it a try next time you need to peek under the hood!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!