Efficient File Uploads in Python with aiohttp

Uploading files is a common task in many web applications. Python's aiohttp library provides useful tools for handling file uploads efficiently in asynchronous, non-blocking server applications.

Uploading Files with aiohttp Web Servers

aiohttp is often used for building high-performance async web servers and APIs in Python. When a client sends a file upload request to an aiohttp server, the file data is available in the request object's payload attribute as a stream of bytes.

Here is an example aiohttp server route that handles file uploads:

import aiohttp
from aiohttp import web

async def handle_file_upload(request):
    data = await request.post()
    file = data['file']
    filename = file.filename
    file_bytes = file.file.read() # read the bytes
    # save contents somehow...
    return web.Response(text="Uploaded file "+filename)

app = web.Application()
app.add_routes([web.post('/upload', handle_file_upload)])

This reads the uploaded file data as a stream of bytes for further processing.

Uploading Large Files

When uploading very large files, we don't want to load the entire file contents into memory at once. This can cause performance issues or even crash our Python process if the file is too large.

Instead, we can process the upload data as a stream, reading small chunks at a time:

CHUNK_SIZE = 4096

async def stream_file_upload(request):
    data = await request.post()
    file = data['file']
    
    with open(file.filename, 'wb') as f:
        chunk = await file.read(CHUNK_SIZE) 
        while chunk:
            f.write(chunk)
            chunk = await file.read(CHUNK_SIZE)

    return web.Response(text="Uploaded "+file.filename)

This asynchronously reads the file upload stream in 4KB chunks, writing each chunk to disk before fetching the next chunk. This prevents large files from consuming too much memory.

File Size and Type Checking

For security and validation, we likely want to verify file sizes and check the file types before handling the upload. aiohttp provides this data in the file headers:

async def validated_upload(request):
    data = await request.post() 
    file = data['file']

    # File size check
    MAX_FILE_SIZE = 1024 * 1024 * 5 # 5MB
    if file.size > MAX_FILE_SIZE:
        return web.Response(text="File too large")

    # File type check
    allowed_types = ['.jpg', '.png']
    ext = file.filename.lower().split('.')[-1]
    if ext not in allowed_types:
        return web.Response(text="Invalid file type")

    # File OK, proceed with upload
    # ....

This checks the file size against a maximum of 5MB, and validates the file extension against a whitelist of allowed types.

Handling Multiple File Uploads

aiohttp also supports uploading multiple files in one request. The uploaded files are available in the request data under the key request.post()['file'] as a list.

We can iterate through and process each one:

async def multiple_files(request):
    data = await request.post()
    
    for file in data['file']:
        filename = file.filename
        # Process upload
    
    return web.Response(text="Uploaded multiple files")  

This allows users to upload batches of files in a single request.

Storage Options for Uploaded Files

Once the file data is uploaded, there are different options for storage:

The filesystem is easiest, while cloud object stores offer scale and durability. Database storage can enable easier metadata handling.

Handling Uploads with Forms

For browser file uploads, aiohttp can work with file input HTML forms:

<form action="/upload" method="post" enctype="multipart/form-data">
  <input type="file" name="file">  
  <button type="submit">Upload</button>
</form>

The enctype attribute enables multipart form data with file uploads. The server route would access the file using await request.post() as shown in previous examples.

In Summary

With these tips, you can add robust file handling to your aiohttp web apps and APIs!