Stories from the Web Crawling trenches in tradeoffs

Making Python Requests Without Timeout

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests library, timeouts are set by default. However, sometimes you may want to remove the timeout to let long requests run to completion.

Python Threads vs Processes: Which is Faster and When to Use Each

Author: Mohan Ganesan

Date: Mar 24, 2024

When writing Python programs, developers often wonder if it's better to use threads or processes. Processes are generally faster and more robust, but have higher overhead. Threads require less resources to create, but come with their own challenges.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!