Accessing URLs Requiring Authentication with Python's urllib

Feb 6, 2024 ยท 2 min read

When accessing web URLs that require authentication, Python's urllib module provides a simple way to supply credentials and access protected resources. Whether you need to pull data from a website or API endpoint, urllib handles the basic auth handshake automatically behind the scenes.

Here's a quick example accessing a protected URL:

import urllib.request
import urllib.parse

username = 'myusername'
password = 'mypassword'

url = ''

p = urllib.parse.urlencode({'username': username, 'password': password})
request = urllib.request.Request(url)
request.add_header('Authorization', 'Basic %s' % p)

response = urllib.request.urlopen(request)
data =

We supply the username and password, encode them into a string using urllib.parse.urlencode, and add an Authorization header to the request with the encoded credentials.

When accessing URLs over HTTPS, this handles the authentication automatically without needing to deal with cookies, sessions, etc.


  • Use a context manager to automatically close the response:
  • with urllib.request.urlopen(request) as response:
       data =
  • If authentication fails, it will raise a HTTPError - catch this for handling invalid creds
  • Scenarios

  • Accessing APIs that require an API key
  • Pulling reports from a web app that requires login
  • Scrape data from a website that uses basic auth protection
  • Using urllib for basic access authentication provides a simple way to supply credentials for restricted URLs. With a few lines of code, you can access protected resources and data.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: