Stories from the Web Crawling trenches in HTTPS requests

Troubleshooting SSL Certificate Errors with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When using Python Requests library for HTTPS requests, you may encounter SSL certificate errors. Try updating OS, specifying custom CA bundle, or disabling certificate verification.

Troubleshooting Python Requests Through a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Common problems and solutions when sending requests through a proxy server in Python code.

Troubleshooting Aiohttp Connecting to the Wrong Host

Author: Mohan Ganesan

Date: Mar 3, 2024

When using the aiohttp library in Python, you may occasionally see errors where aiohttp attempts to connect to the wrong host. There are a few things you can try to resolve it: check your DNS configuration, specify the host explicitly, use IP addresses instead of hostnames, and add server name indication (SNI) for HTTPS connections.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!