Logging and Debugging with Requests

Oct 22, 2023 ยท 6 min read

Introduction

Working with HTTP requests in Python often means using the excellent Requests library. It provides a simple interface for sending requests and handling responses.

However, sometimes you need more visibility into what Requests is doing under the hood. Logging and debugging capabilities become essential for troubleshooting issues or just understanding the full request/response flow.

In this comprehensive guide, we'll explore various techniques for enabling detailed logging and debugging with Requests.

Overview of Logging and Debugging with Requests

The Requests library uses several internal components for handling HTTP connections, including:

  • The urllib3 library for managing connections.
  • The http.client module for low-level HTTP protocol handling.
  • To enable detailed logging, we need to configure both of these components.

    The key goals are:

  • Log full request details - method, URL, headers, body.
  • Log full response details - status, headers, content.
  • Include tracebacks to see call flow.
  • Format logs for readability.
  • Minimal code changes to enable debugging.
  • Requests doesn't have great built-in logging, so we need to dig into the internals to enable it.

    When to Enable Debugging

    You wouldn't want verbose debugging enabled in production. But during development and testing, it becomes indispensable.

    Some common use cases:

  • APIs failing mysteriously - debug requests/responses to get clues.
  • Intermittent connection issues - logging may reveal patterns.
  • Testing a new integration - understand request handling details.
  • General debugging - trace flow and data through system.
  • Debug logs allow offline analysis. You can save them to files and dig in later if issues crop up.

    Enabling logging in your integration tests provides ongoing visibility into API calls.

    Built-in Logging

    Let's start with Requests' built-in logging capabilities.

    Using urllib3 Logger

    The urllib3 library handles HTTP connections in Requests. We can enable debug logs by configuring its logger.

    Importing and Configuring Logger

    First import the logger and set the log level to DEBUG:

    import logging
    log = logging.getLogger('urllib3')
    log.setLevel(logging.DEBUG)
    

    This will enable detailed debug logging from urllib3.

    Setting Log Level

    To further refine the logging, you can set the severity threshold.

    For example, to show warnings and above:

    log.setLevel(logging.WARNING)
    

    The hierarchy from most verbose to least is:

  • DEBUG
  • INFO
  • WARNING
  • ERROR
  • CRITICAL
  • Printing Requests and Responses

    With debug logging enabled, you can see the full request and response details:

    DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): example.com:80
    DEBUG:urllib3.connectionpool:<http://example.com:80> "GET / HTTP/1.1" 200 3256
    

    This prints the method, URL, protocol, status code, and response size.

    Setting HTTPConnection Debug Level

    For even more verbosity, we can enable debugging in the http.client module that urllib3 uses under the hood:

    Importing HTTPConnection

    Import HTTPConnection from http.client:

    from http.client import HTTPConnection
    

    Setting debuglevel

    Set the debuglevel to 1 to enable debugging:

    HTTPConnection.debuglevel = 1
    

    Example Debug Output

    Now debugs will show the request headers and body:

    DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): example.com:80
    DEBUG:send: b'GET / HTTP/1.1
    Host: example.com
    User-Agent: python-requests/2.22.0
    
    '
    DEBUG:reply: 'HTTP/1.1 200 OK\\r\\n'
    DEBUG:header: Server: nginx
    DEBUG:header: Content-Type: text/html
    DEBUG:header: Content-Length: 1234
    DEBUG:urllib3.connectionpool:<http://example.com:80> "GET / HTTP/1.1" 200 1234
    

    This gives us complete request/response details from the built-in logging.

    Basic Logging Configuration

    For more control over logging, we can configure it directly instead of using the built-in options.

    Importing Logging Module

    Import Python's logging module:

    import logging
    

    This will let us configure loggers and handlers.

    Setting Root Logger Level

    Set the global log level on the root logger:

    logging.basicConfig(level=logging.DEBUG)
    

    This enables debug logging globally.

    Printing Debug Messages

    We can now print log messages:

    logger = logging.getLogger(__name__)
    
    logger.debug("Request headers: %s", headers)
    logger.debug("Request body: %s", body)
    

    The messages will be emitted because of the debug log level.

    Custom Logging

    For more advanced usage, we can customize formatting, utilize hooks, and route logs.

    Formatted Output

    Basic logging just prints plaintext messages. For readability, we can format the output.

    Creating Custom Formatter

    Define a formatter class to add structure:

    class RequestFormatter(logging.Formatter):
    
        def format(self, record):
            message = super().format(record)
            # Add custom formatting
            return message
    

    Configuring Handler and Formatter

    Next configure a handler to use the formatter:

    handler = logging.StreamHandler()
    handler.setFormatter(RequestFormatter())
    

    Printing Formatted Logs

    Logging through the handler will now be formatted:

    logger = logging.getLogger(__name__)
    logger.addHandler(handler)
    
    logger.info("Formatted request log")
    

    Logging Hooks

    Hooks allow tapping into Requests before and after requests. We can use them to log.

    Request Hook

    Define a hook function to log on requests:

    def log_request(request):
        logger.debug("Request: %s", request)
    

    Response Hook

    Similarly, log after responses:

    def log_response(response):
        logger.debug("Response: %s", response)
    

    Request-Response Roundtrip

    Putting it together to log the full roundtrip:

    session = requests.Session()
    session.hooks["request"].append(log_request)
    session.hooks["response"].append(log_response)
    

    Printing via Hooks

    Now any requests through this session will be logged:

    session.get("<http://example.com>")
    

    This generates:

    Request: <PreparedRequest [GET]>
    Response: <Response [200]>
    

    Hooks give us fine-grained control to log events.

    Advanced Configuration

    There are a few other useful logging configurations.

    Logging to File

    For persistent storage, write logs to a file:

    file_handler = logging.FileHandler("debug.log")
    logger.addHandler(file_handler)
    

    This will save logs instead of printing to stdout.

    Logging to Stdout

    To print to standard output instead of stderr:

    stdout_handler = logging.StreamHandler(sys.stdout)
    logger.addHandler(stdout_handler)
    

    Disabling Logging

    Disable debugging with:

    logger.setLevel(logging.WARNING)
    

    Increase the threshold to suppress less important messages.

    Third-Party Libraries

    For more advanced logging, consider dedicated libraries like loguru. They provide much easier configuration and formatting.

    Conclusion

    Summary of Techniques

    In this guide, we covered various techniques for debugging Requests:

  • Enabling urllib3 and http.client debugging
  • Configuring basic logging
  • Custom formatting and hooks
  • Writing logs to files or stdout
  • Advanced handling with 3rd party libraries
  • Recommendations

  • Use urllib3 and HTTPConnection for quick debugging.
  • Employ custom formatting and hooks for production logging.
  • Send logs to files for offline analysis.
  • Try advanced loggers like loguru if you need more capabilities.
  • Careful logging will provide invaluable visibility into your Requests usage. Following these patterns will ensure you have the right level of verbosity when issues arise or you need to audit request handling in your systems.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!