Stories from the Web Crawling trenches in connection issues

Troubleshooting aiohttp ServerDisconnectedError

Author: Mohan Ganesan

Date: Feb 22, 2024

If you're using Python's aiohttp library for asynchronous HTTP requests and getting ServerDisconnectedErrors, here are some troubleshooting tips to handle the response inside the context manager and check for connectivity issues.

Troubleshooting Python Request Timeouts

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python can sometimes result in timeouts due to slow network connection, overloaded API servers, short timeout values, or connection issues. To handle timeouts, you can check connectivity, increase the timeout duration, implement retries, handle exceptions, and assess for overload. Best practices to avoid timeouts include monitoring requests, stress testing remote APIs, implementing circuit breakers, and caching API response data.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!