Passing Parameters in URLs with Python's urllib

Feb 8, 2024 ยท 2 min read

When constructing URL requests in Python, you'll often need to append parameters to the URL to pass additional data to the server. The urllib module provides easy ways to handle this.

For example, say we need to call a URL like:

https://api.example.com/data?category=science&format=json

This passes two parameters, category and format, to the server. Here's how to construct that in Python:

import urllib.parse
import urllib.request

base_url = 'https://api.example.com/data'
params = {'category': 'science', 'format': 'json'}

url = '{}?{}'.format(base_url, urllib.parse.urlencode(params))

data = urllib.request.urlopen(url).read()

The key points:

  • We construct a dictionary params containing the parameters we want to pass
  • We use urllib.parse.urlencode() to encode the dictionary in a URL-safe format
  • We append the encoded parameters to the base URL to construct the full URL
  • We can then use urllib.request.urlopen() to send the request
  • Some tips:

  • Always URL-encode parameter values in case they contain special characters
  • Use a dictionary to track multiple parameters, then encode the whole thing at once
  • You can also pass the params dictionary directly to urlopen() and it will encode it
  • For GET requests, params go in the URL; for POSTs, they would go in the request body
  • Overall, urllib makes it straightforward to pass parameters around. The parse module handles all the encoding details for you. And you can use the standard dictionary methods to easily construct, combine, and manage multiple parameters to pass along.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!