Stories from the Web Crawling trenches in timeouts

How to Handle Timeout error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Timeouts are critical for making requests in Python. They prevent hanging requests and wastage of resources. The requests library provides flexible ways to set timeouts globally or per-request.

Using httpx's AsyncClient for Asynchronous HTTP POST Requests

Author: Mohan Ganesan

Date: Feb 5, 2024

The httpx library in Python provides an AsyncClient class that makes it easy to send asynchronous HTTP requests without having to deal with some of the complexity of asyncio directly.

Handling Timeouts Gracefully with aiohttp in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

When building asynchronous web applications and APIs in Python with aiohttp, properly handling timeouts is essential. Use ClientTimeout to configure request timeouts and wrap requests in try/except blocks to catch ClientTimeout. Configure global timeout on aiohttp servers with timeout parameter.

Handling Timeouts Gracefully with Python's urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

When fetching data from external websites and APIs, handling timeouts gracefully and implementing retry logic with exponential backoff is crucial for building robust applications.

Making Python Requests Without Timeout

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests library, timeouts are set by default. However, sometimes you may want to remove the timeout to let long requests run to completion.

Making Asynchronous HTTP Requests in Discord.py

Author: Mohan Ganesan

Date: Mar 3, 2024

Discord bots built with discord.py library can run multiple actions in parallel using aiohttp for asynchronous HTTP requests.

Troubleshooting Slow and Failing Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python applications can sometimes be problematic. This article provides tips for troubleshooting slow or failing requests, including checking for network/server issues, setting sensible timeouts, inspecting the request object, and profiling long requests.

Making HTTP Requests in Python Without a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Make HTTP requests in Python without a proxy using the requests library. Customize requests with headers, parameters, and handle timeouts.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!