Simplifying HTTP Requests in Python with urllib

Feb 3, 2024 ยท 2 min read

The urllib module in Python provides useful tools for fetching data from the web. With just a few lines of code, you can make HTTP requests and parse responses without needing to use an external library.

Making GET Requests

To make a basic GET request, you first need to import urllib and call urllib.request.urlopen() with the URL you want to fetch:

import urllib.request

with urllib.request.urlopen('') as response:
   html =

The urlopen() function returns a file-like object that you can read from. This saves you having to worry about low-level details like opening a socket.

Handling HTTP Responses

The response object has various useful methods and attributes for checking the status and handling the data:

print(response.status) # prints the status code e.g. 200
print(response.headers) # print the response headers
html = # read the response body

So with very little code, you can print out status codes, headers, and fetch the HTML content.

Constructing Requests

For more advanced usage, you can construct a Request object and customize details like headers and method:

request = urllib.request.Request('', 
                 headers={'User-Agent': 'My Program'})
response = urllib.request.urlopen(request)

Here we added a custom user-agent header to identify our program. This gives you full flexibility to tweak requests before sending.

Overall, urllib strikes a great balance between simplicity and control. Give it a try for your next Python project that needs to talk to web APIs or scrape websites!

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl ""

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: