Stories from the Web Crawling trenches in HTTPS

How to fix SSLError in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Properly handle SSL errors in Python requests by updating CA bundles, fixing certificates, and using TLS 1.2+. Use SSLContext for full control over SSL behavior.

Accessing HTTPS Sites with Self-Signed Certs in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Methods to securely access HTTPS sites using self-signed certificates with Python Requests: certifi bundle, custom PEM certs, REQUESTS_CA_BUNDLE, SSLContext.

Using Proxies with Python Requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Python requests library simplifies HTTP requests and API calls. Proxies help avoid IP blocking. Configure proxies using a dictionary or environment variables. Authenticate requests with credentials. Use sessions for persistent data. Disable SSL verification if trusted. Adjust timeouts and retries for robust requests.

Secure HTTP Requests in Python with aiohttp ClientSession SSL

Author: Mohan Ganesan

Date: Feb 22, 2024

Making secure HTTPS requests in Python simplified with aiohttp ClientSession SSL functionality.

Making Secure HTTP Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Python requests library makes HTTPS requests simple and secure, providing easy syntax, encryption, validation, and access to response data.

urllib certificate verify failed

Author: Mohan Ganesan

Date: Feb 6, 2024

urllib in Python may encounter SSL certificate verification errors. Try checking for expired certificates, disabling certificate verification, updating certificates, and using certificate pinning.

Accessing Websites in Python with urllib.request.urlopen

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib.request module in Python 3 provides a simple way to access and download data from websites via HTTP and HTTPS.

Making HTTPS Requests in Python with Requests and Certifi

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTPS requests in Python, it's important to have SSL/TLS certificate verification enabled to ensure secure connections.

Making API Requests Safely with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making API requests in Python, it's important to consider security. Use HTTPS, validate certificates, use tokens for authentication, and handle sensitive data safely.

How to Use cURL with a Proxy

Author: Mohan Ganesan

Date: Apr 30, 2024

cURL is a powerful command-line tool for transferring data using proxies. It supports HTTP, HTTPS, and SOCKS proxies, and allows for data extraction and proxy authentication.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!