Efficient File Uploads in Python with aiohttp

Feb 22, 2024 ยท 4 min read

Uploading files is a common task in many web applications. Python's aiohttp library provides useful tools for handling file uploads efficiently in asynchronous, non-blocking server applications.

Uploading Files with aiohttp Web Servers

aiohttp is often used for building high-performance async web servers and APIs in Python. When a client sends a file upload request to an aiohttp server, the file data is available in the request object's payload attribute as a stream of bytes.

Here is an example aiohttp server route that handles file uploads:

import aiohttp
from aiohttp import web

async def handle_file_upload(request):
    data = await request.post()
    file = data['file']
    filename = file.filename
    file_bytes = file.file.read() # read the bytes
    # save contents somehow...
    return web.Response(text="Uploaded file "+filename)

app = web.Application()
app.add_routes([web.post('/upload', handle_file_upload)])

This reads the uploaded file data as a stream of bytes for further processing.

Uploading Large Files

When uploading very large files, we don't want to load the entire file contents into memory at once. This can cause performance issues or even crash our Python process if the file is too large.

Instead, we can process the upload data as a stream, reading small chunks at a time:


async def stream_file_upload(request):
    data = await request.post()
    file = data['file']
    with open(file.filename, 'wb') as f:
        chunk = await file.read(CHUNK_SIZE) 
        while chunk:
            chunk = await file.read(CHUNK_SIZE)

    return web.Response(text="Uploaded "+file.filename)

This asynchronously reads the file upload stream in 4KB chunks, writing each chunk to disk before fetching the next chunk. This prevents large files from consuming too much memory.

File Size and Type Checking

For security and validation, we likely want to verify file sizes and check the file types before handling the upload. aiohttp provides this data in the file headers:

async def validated_upload(request):
    data = await request.post() 
    file = data['file']

    # File size check
    MAX_FILE_SIZE = 1024 * 1024 * 5 # 5MB
    if file.size > MAX_FILE_SIZE:
        return web.Response(text="File too large")

    # File type check
    allowed_types = ['.jpg', '.png']
    ext = file.filename.lower().split('.')[-1]
    if ext not in allowed_types:
        return web.Response(text="Invalid file type")

    # File OK, proceed with upload
    # ....

This checks the file size against a maximum of 5MB, and validates the file extension against a whitelist of allowed types.

Handling Multiple File Uploads

aiohttp also supports uploading multiple files in one request. The uploaded files are available in the request data under the key request.post()['file'] as a list.

We can iterate through and process each one:

async def multiple_files(request):
    data = await request.post()
    for file in data['file']:
        filename = file.filename
        # Process upload
    return web.Response(text="Uploaded multiple files")  

This allows users to upload batches of files in a single request.

Storage Options for Uploaded Files

Once the file data is uploaded, there are different options for storage:

  • Filesystem - Write files to a disk folder or mount
  • Object Storage - Upload to S3, Google Cloud Storage, etc
  • Database - Store file bytes in a BLOB database column
  • The filesystem is easiest, while cloud object stores offer scale and durability. Database storage can enable easier metadata handling.

    Handling Uploads with Forms

    For browser file uploads, aiohttp can work with file input HTML forms:

    <form action="/upload" method="post" enctype="multipart/form-data">
      <input type="file" name="file">  
      <button type="submit">Upload</button>

    The enctype attribute enables multipart form data with file uploads. The server route would access the file using await request.post() as shown in previous examples.

    In Summary

  • aiohttp provides a straightfoward API for handling file uploads from clients
  • Carefully validate and process uploads as byte streams rather than full file loads
  • Check file headers for size/type before storage to prevent abuse
  • Support multiple parallel uploads for improved user experience
  • Store uploaded files appropriately based on application needs
  • With these tips, you can add robust file handling to your aiohttp web apps and APIs!

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: