Streaming Uploads in Python Requests using File-Like Objects

Feb 3, 2024 ยท 2 min read

When making requests in Python, you may sometimes need to upload large files like videos, audio, or other binary data. Rather than loading the entire file contents into memory, you can upload data in chunks using a file-like object.

This saves memory and allows streaming uploads even for very large files.

Why Use File-Like Objects?

Uploading data using the files argument in Requests loads the full contents of the file into memory. This can cause problems for large files:

with open('large_video.mp4', 'rb') as f:'', files={'video': f}) 

Instead, we can create a file-like object that streams the data in chunks:

with open('large_video.mp4', 'rb') as f:'', data=f)

This streams the file data in memory-efficient chunks without loading the entire file.

Creating a File-Like Object in Python

Many built-in Python file objects already support file-like streaming uploads:

with open('data.bin', 'rb') as f:, data=f)

You can also create a custom file-like object by implementing Python's file-like API:

import io

class StreamingBody:
    def __init__(self, filepath):
        self.filepath = filepath
    def read(self, n):
        # custom read implementation

    def seek(self, offset):
        # custom seek implementation

stream = StreamingBody('video.mp4'), data=stream)

When to Avoid Streaming Uploads

Streaming file uploads add complexity and may not always be necessary. For smaller files under 10-20MB, avoid complex file streams.

Just loading the contents directly with the files argument works great:

with open('report.pdf', 'rb') as f:, files={'report': f})

So in summary, file-like streaming objects allow efficient upload of large binary data in Python Requests while avoiding high memory usage. Implement the file-like API on any object to enable streaming uploads.

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl ""

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: