Sending Multipart Form Data with Python's urllib

Feb 8, 2024 ยท 3 min read

When building Python applications that integrate with web services, you may need to send multipart form data, which contains both file uploads and regular form data. Python's standard urllib library provides the tools to handle this.

What is Multipart Form Data?

Multipart form data is used in web forms that have file uploads. Along with the file data, the form also sends regular form data like text fields. The form data is sent in multiple "parts" - one part for the file data, another part for each text field.

The web server reconstructs the parts on the receiving end. This allows files and data to be transmitted together in one request.

Creating a Multipart Form Encoder

Python's urllib library provides the MultipartEncoder to build this multipart data:

import urllib

m = urllib.MultipartEncoder(
    fields={
         # regular form fields
         "text_field": "text value",
         "other_field": "other value",
         # the upload file 
         "file_upload": ("filename", open("file.txt", "rb"), "text/plain")
    }
)

The fields parameter is a dictionary mapping field names to values. For file uploads, use a 3-item tuple with the filename, file handle, and file content type.

Setting the Request Content-Type

When sending multipart form data, the request must include the Content-Type header with the boundary value from the encoder:

import requests

url = "https://api.example.com/upload"

r = requests.post(url, 
    data=m,
    headers={"Content-Type": m.content_type})

The server uses this boundary to distinguish between each part of the multipart data.

Posting Multipart Forms with Requests

The requests library can simplify sending multipart form data. You can directly pass the MultipartEncoder as the data parameter. requests will automatically add the proper Content-Type header:

import requests

url = "https://api.example.com/upload"

r = requests.post(url, data=m) 

Handling Large Uploads

For large file uploads, multipart encoding can consume a lot of memory. An alternative is to send the file data in chunks:

with open("big_file.dat") as f:
    while True:
       chunk = f.read(1024)
       if not chunk: 
           break
       requests.post(url, data=chunk)

This streams the upload directly from disk rather than loading the entire file into memory.

Key Takeaways

  • Use Python's MultipartEncoder to bundle file and form data
  • Set the Content-Type header to include the encoder boundary
  • requests automatically handles multipart data and headers
  • Stream file uploads to reduce memory usage
  • The urllib and requests libraries provide the framework for sending robust multipart form data requests from Python. With some care to handle large files, you can build applications that reliably upload files and data to web services.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!