Stories from the Web Crawling trenches in connection pooling

Using httpx's AsyncClient for Asynchronous HTTP POST Requests

Author: Mohan Ganesan

Date: Feb 5, 2024

The httpx library in Python provides an AsyncClient class that makes it easy to send asynchronous HTTP requests without having to deal with some of the complexity of asyncio directly.

Mastering Python Requests Sessions for Power Users

Author: Mohan Ganesan

Date: Oct 22, 2023

The Python requests library provides a powerful Session object for handling HTTP requests. Sessions allow you to persist settings, reuse connections, and handle cookies automatically.

Keeping Sessions Alive with Persistent Connections in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Using persistent sessions in Python Requests library improves performance and allows reusing connections for multiple requests.

Sending Form Data with Python Requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Sending form data is a common task in web development. Learn how to do it effectively with Python Requests library.

Making Asynchronous HTTP Requests in Python with aiohttp Connectors

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library provides a powerful tool for making asynchronous HTTP requests in Python. The aiohttp.TCPConnector manages connection pooling and reuse, allowing for improved performance and optimization of HTTP clients and services.

Making the Most of aiohttp's TCPConnector for Asynchronous HTTP Requests

Author: Mohan Ganesan

Date: Mar 3, 2024

Carefully configuring aiohttp's TCPConnector is key to get the most out of asynchronous HTTP in Python.

Do I need urllib3?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python's urllib3 module provides connection pooling, asynchronous requests, and TLS encryption for better performance in HTTP requests.

Why use urllib3?

Author: Mohan Ganesan

Date: Feb 20, 2024

urllib3 is a full-featured HTTP client for making requests in Python. It handles connection pooling, SSL/TLS verification, and more, making it a popular choice for web APIs.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!