When accessing HTTPS URLs in Python using the Requests module, you may encounter SSL certificate verification errors if the site uses a self-signed certificate rather than one signed by a trusted certificate authority. This poses a challenge for testing and development scenarios.
Fortunately, the Requests module provides options to handle these self-signed certificate cases in a secure way. Here we'll explore a few methods to safely access HTTPS sites that use self-signed certificates with Requests.
The Risks of Ignoring SSL Errors
First it's important to understand why manually disabling SSL certificate verification is risky. Doing something like:
This exposes your application to man-in-the-middle (MITM) attacks where data can be intercepted or manipulated. So disabling verify is not recommended in production.
Using certifi to Load Root Certificates
The easiest way to handle self-signed certs is including the certifi package along with Requests. This contains root certificates from Mozilla to validate certificate chains.
Just install certifi and Requests will use it automatically:
pip install requests certifi
This allows Requests to validate certificate chains while still allowing you to access sites with self-signed leaf certificates.
Supplying Your Own Certificate Bundle
Alternatively, you can provide your own custom bundle containing the self-signed cert:
cert_path = '/path/to/self-signed-cert.pem'
This is useful for cases where you want to test with fully custom certs not covered by root stores like certifi.
Using REQUESTS_CA_BUNDLE Environment Variable
This approach sets the trusted certs globally without coding it for each script.
Wrapping the SSL Context
For more advanced cases, you can wrap the default SSL context to add certs or adjust other SSL options:
ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile='/path/to/self-signed-cert.pem')
requests.get('https://self-signed.example.com', verify=False, ssl_context=ssl_context)
Here we handle the cert verification at the SSL context level.
Development Server Certificates
For development/testing it's also common to use automation tools like Ansible, Puppet or Docker to automatically generate and install certs on local servers to avoid manual steps.
These tools typically allow exporting the CA certificate to distribute to clients. You can feed this CA cert bundle into one of the above Requests options to complete the chain of trust.
In this article we covered several methods to securely access HTTPS-enabled sites using self-signed certificates with Python Requests:
Taking advantage of these options allows accessing HTTPS test servers during development while still maintaining security.
The key takeaway is to always validate certificate chains rather than disabling verification outright. This prevents man-in-the-middle attacks when working with self-signed certs.