Debugging HTTP Requests with httpx Debug

Feb 5, 2024 ยท 4 min read

Making HTTP requests is core functionality for many Python applications. Whether fetching data from an API or interacting with a web service, HTTP clients like requests and httpx enable this crucial communication.

But when an HTTP request isn't working as expected, debugging the root cause can be tricky. This is where httpx debug comes in handy.

What is httpx debug?

The httpx_debug package provides a debugging proxy server that captures HTTP traffic between your application and external servers. By running your app through this local proxy, you gain visibility into the raw requests and responses to identify issues.

Some key capabilities provided by httpx debug:

  • Logs all request/response data to the terminal or to a file
  • Lets you mock responses for testing scenarios
  • Modifies requests and responses for testing edge cases
  • Saves sessions to disk for later inspection
  • So if you've ever spent too long printing and parsing HTTP calls or struggled to recreate obscure bugs, httpx debug takes much of the pain out of debugging.

    Getting Started with the Debug Proxy

    Install httpx debug via pip:

    pip install httpx_debug

    Then launch the debug proxy from the command line:

    httpx-debug proxy

    This starts the proxy server on port 9091.

    Now run your application code while pointing at the proxy URL - for example:

    import httpx
    
    client = httpx.Client(proxies="http://127.0.0.1:9091") 
    resp = client.get("https://example.com")
    print(resp.text)

    The proxy will intercept and print information on the requests/responses made through it:

    Request: 1 > GET https://example.com/
    Response: 1 < 200 OK 

    This output shows the request method, URL, and response status for each call.

    Inspecting Full Request/Response Details

    To see full request/response data, enable debug mode on the proxy:

    httpx-debug proxy --debug

    Now details like headers, bodies, and timing are included:

    Request 1 > GET https://example.com
    Headers: {'User-Agent': 'python-httpx/0.22.0'}
    
    Response 1 < 200 OK 
    Headers: {'Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '1256'}
    Body: <html>
          <head>...</head>
    ...

    This makes it easy to inspect exactly what your application is sending and receiving.

    Modifying Traffic with Transform Plugins

    One powerful feature of httpx debug is its plugins for transforming requests and responses. Plugins can simulate error conditions and edge cases.

    For example, to induce slow responses, use the delay plugin:

    httpx-debug proxy --plugin delay

    This will pause each response by one second. Your app code can then be tested for timeout issues.

    There are also plugins for throttling bandwidth, dropping packets, and more. See the httpx debug docs for details on all included plugins.

    Mocking Upstream Responses

    Need to test your code without calling real services? Httpx debug lets you easily mock upstream responses.

    Use the --mock flag when starting the proxy. This will return 404 responses from all unknown hosts:

    httpx-debug proxy --mock

    Now requests to non-recognized hosts will receive mock 404 responses rather than calling real servers.

    Take control of mock data by creating YAML files defining custom responses. For example:

    # mock-data.yml
    - request:
        url: "https://api.example.com/users" 
      response:
        status_code: 200
        headers:
          Content-Type: application/json
        json:
          - id: 1
            name: Alice
          - id: 2 
            name: Bob

    With this file available, mock responses can be tailored as needed.

    Debugging HTTPS Traffic

    By default, the debug proxy handles only plaintext HTTP traffic. To capture HTTPS, enable man-in-the-middle mode using the builtin Mitmproxy certificate authority:

    httpx-debug mitm

    This will install a trusted MITM cert on your local machine allowing the proxy to decrypt SSL traffic.

    With MITM enabled, the proxy can debug secure requests just like plaintext HTTP.

    Final Tips

    Httpx debug brings visibility into your HTTP calls that is hard to achieve otherwise. Here are some final tips:

  • Save request/response logs to disk with --log-file=output.log for later review
  • Share saved session files with others to replicate issues
  • Try different plugins like delay and throttle to test edge cases
  • Mock upstream services with custom YAML files when testing
  • Debugging HTTP issues in complex applications can be frustrating. By running traffic through httpx debug, you gain insight to fix problems faster and build more resilient systems.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!