Making Scheme-Agnostic HTTP Requests in Python

Feb 3, 2024 ยท 2 min read

When making HTTP requests in Python using the popular requests library, the scheme (http:// or https://) is usually specified in the URL. However, sometimes you may want to make a request in a scheme-agnostic way without hardcoding http or https.

This can be useful in situations where:

  • You want to support both HTTP and HTTPS connections depending on some runtime configuration
  • You are making requests to internal services that use custom schemes
  • You want to simplify your code by not having to specify the scheme repeatedly
  • Fortunately, the requests library makes this easy to do.

    Constructing Scheme-Agnostic URLs

    Instead of hardcoding the scheme like:

    url = ""

    You can omit it:

    url = "//"

    This constructs a scheme-relative URL that will use the default scheme determined by the request context at runtime.

    Making Requests

    Once you have your scheme-agnostic URL, making the actual request works the same:

    import requests
    url = "//"
    response = requests.get(url, timeout=5)

    The request will default to either HTTP or HTTPS based on factors like if the host URL supports HTTPS, environment variables, etc.

    Wrap Up

    Constructing scheme-relative URLs and allowing requests to handle the scheme selection simplifies your code and enables greater flexibility. Give it a try next time you want your Python application to support both HTTP and HTTPS connections!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!