Fixing Memory Leaks in Python requests

Feb 3, 2024 ยท 2 min read

Python's requests library makes sending HTTP requests simple and convenient. However, one common issue developers face is memory leaks when using requests. This happens when requests keeps references to response objects in memory unnecessarily, preventing garbage collection.

Let's walk through a simple example to understand what's happening under the hood.

import requests

response = requests.get('') 

Here, we make a GET request and assign the response to response. Behind the scenes, requests stores the response content in memory. If we don't explicitly close or delete the response, that content will occupy memory indefinitely.

Over time, making repeated requests like this causes the process memory usage to grow continually - a memory leak.

The Solution - Close Connections

The easiest fix is closing the connection after we are done with the response by calling response.close(). This informs requests that we no longer need the response, allowing it to clean up the connection and free memory.

import requests

response = requests.get('')
# Process response


Alternatively, we can use a context manager to automatically close connections:

with requests.get('') as response:
   # Process response

This ensures connections get closed promptly even if exceptions occur.

Other Recommendations

  • Use connection pools for making multiple requests as they reuse connections.
  • Avoid caching entire response contents, extract only what you need.
  • Use streaming response processing for large responses.
  • Following best practices like these prevents accidental memory leaks and makes requests usage efficient. The key is being mindful about cleaning up unneeded response data.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: