Handling 404 Errors when Making HTTP Requests in Python

Feb 3, 2024 ยท 2 min read

When making HTTP requests in Python, you may occasionally run into 404 errors, indicating that the URL you tried to request does not exist on the server. Here are some tips on handling these "page not found" errors gracefully in your Python code.

Check for 404 Errors

After making a request using the requests module, check the response status code before trying to use the response content:

import requests

response = requests.get("http://example.com/path")
if response.status_code == 404:
    # Handle 404 error
    print("URL does not exist") 
    # Use response content

This checks if the status code in the response is 404 before processing the content.

Log and Notify

Instead of just printing, you may want to log these 404 errors to help debug issues later:

import logging

logger = logging.getLogger(__name__) 

logger.warning("Got 404 for URL %s", url)

You can also notify someone through email or Slack that a URL is returning 404.

Use a Try-Except Block

Wrap your request in a try-except block to handle exceptions gracefully:

    response = requests.get(url)
except requests.exceptions.HTTPError as err:
    if err.response.status_code == 404:
        print("404 Error") 

This avoids your program crashing if a URL returns 404.

Handling 404s properly ensures your program does not break unexpectedly. Log and monitor these errors to fix invalid URLs in your code.

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: