Stories from the Web Crawling trenches in long requests

Making Python Requests Without Timeout

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests library, timeouts are set by default. However, sometimes you may want to remove the timeout to let long requests run to completion.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!