A Beginner's Guide to Uploading Files with Python Requests

Oct 31, 2023 ยท 5 min read

Requests is a popular Python library that makes it easy to send HTTP requests. This is helpful for web scraping and interacting with APIs. One common task is uploading files via a POST request. In this guide, I'll walk through how to do this in Requests step-by-step.

An Introduction to Requests

First, what exactly is Requests? It's a Python module that provides a simple interface for making HTTP calls. The key benefit is that it abstracts away a lot of the low-level details of dealing with HTTP requests and responses for you.

Some key features of Requests:

  • Simple APIs for common HTTP methods like GET, POST, PUT, DELETE
  • Automatic JSON encoding and decoding
  • Support for forms and file uploads
  • Connection pooling and sessions
  • Handy utilities like timeout, retries, redirects
  • Requests makes it super easy to do things like scrape web pages, interact with APIs, upload files, and more!

    Our Goal

    In this guide, I'm going to walk through the process for uploading files via Requests, step-by-step. We'll start simple with a single file upload, then look at uploading multiple files in one go. I'll also cover some more advanced options you may need to use.

    My goal is to provide you with a solid understanding of the key concepts so you can apply them in your own projects. Let's get to it!

    Uploading a Single File

    Uploading a single file with Requests is straightforward. There are just a few steps we need to follow:

    Step 1: Open the File and Get the Bytes

    First, we need to open the file we want to upload and read its bytes. This is so Requests can determine the body and content-length to send.

    We use the built-in open() function for this. Pass it the filename and "rb" mode to get bytes:

    f = open("file.jpg", "rb")
    file_bytes = f.read()
    

    The "b" is key - it returns bytes which Requests needs.

    Step 2: Create a Files Dict with the Bytes

    Next, we need to create a dictionary with the file bytes to pass to Requests. This tells it what to upload.

    I like to structure it as a tuple with the filename, bytes, and optional content-type:

    files = {
        "myfile": ("file.jpg", file_bytes, "image/jpeg")
    }
    

    The key can be any field name you want.

    Step 3: Make the POST Request

    Now we can make the POST request, passing the files dict:

    url = "<https://api.example.com/upload>"
    
    r = requests.post(url, files=files)
    

    This will upload the file for us. All we had to do was open the file and provide the bytes!

    Step 4: Check the Response

    Finally, we should check the response status code to confirm the upload succeeded:

    if r.status_code == 201:
        print("Upload successful!")
    else:
        print("Upload failed!")
    

    And that's it for a basic single file upload with Requests! Pretty easy right?

    Now let's look at uploading multiple files...

    Uploading Multiple Files

    To upload multiple files in a single request, we just need to add additional key/value pairs to our files dictionary.

    For example:

    files = {
      "file1": ("report.pdf", pdf_bytes),
      "file2": ("presentation.pptx", pptx_bytes)
    }
    

    Then pass this to .post() like before:

    r = requests.post(url, files=files)
    

    Requests will upload all the files concurrently. Awesome!

    Some tips when uploading multiple files:

  • You can reuse field names if needed
  • Try a list of tuples as an alternative format
  • Check responses for each file
  • And that's the basics of multiple file uploads! Next let's look at some additional options...

    Additional Options

    There are a few other options you may need to use when uploading files:

    Headers

    You can pass custom headers for things like content-type and authentication:

    headers = {"Authorization": "Bearer mytoken"}
    
    requests.post(url, files=files, headers=headers)
    

    Data and JSON

    You can also send data and JSON bodies alongside your files:

    data = {"key": "value"}
    json = {"key": "value"}
    
    requests.post(url, files=files, data=data, json=json)
    

    Note that json overrides data if both are passed.

    Redirects

    By default Requests will follow redirects, but you can disable this:

    requests.post(url, files=files, allow_redirects=False)
    

    Timeouts

    Use the timeout parameter to avoid hanging requests:

    requests.post(url, files=files, timeout=3.05)
    

    Authentication

    Pass HTTP Basic Auth or Digest Auth tuples:

    requests.post(url, files=files, auth=('user','pass'))
    

    This covers some of the key options you may need to use alongside file uploads.

    Handling Errors

    When uploading files, errors can occur like:

  • 400 Bad Request - Invalid parameters
  • 403 Forbidden - Authentication failure
  • 413 Payload Too Large - File is too big
  • Always check the status code to catch errors.

    For example:

    r = requests.post(url, files=files)
    
    if 400 <= r.status_code < 500:
      print("Client error:", r.text)
    elif 500 <= r.status_code < 600:
      print("Server error")
    

    This prints custom error messages based on the status code.

    You can also use try/except blocks to handle specific exceptions nicely:

    try:
      r = requests.post(url, files=files, timeout=10)
      r.raise_for_status()
    
    except requests.Timeout as e:
      print("Timeout occurred, try again")
    
    except requests.HTTPError as e:
      print("HTTP error occurred:", e)
    

    This lets you handle timeouts and HTTP errors independently.

    Debugging file uploads can be tricky, so take time to handle errors properly in your code.

    Recap and Key Takeaways

    Let's recap what we learned:

  • Open files and get bytes to upload
  • Create a files dict with tuples
  • Pass files to .post()
  • Handle errors and check status codes
  • Set headers, data, timeouts as needed
  • The key points are:

  • Get file bytes to pass to Requests
  • Use the files parameter on .post()
  • Always handle errors and check responses
  • That covers the basics and best practices of uploading files with Python Requests!

    Next Steps

    To dive deeper:

  • Read the Requests documentation
  • Learn how Requests handles streaming uploads
  • Try using the Requests session object for efficiency
  • I hope you found this guide helpful! Let me know if you have any other questions.

    Now you're ready to start uploading files via Requests in your own projects. Good luck!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!