Stories from the Web Crawling trenches in redirects

How to fix TooManyRedirects error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

The TooManyRedirects error in Python requests occurs when the request exceeds the default limit of 30 redirects. This article explains the causes of the error and provides solutions to fix it, including modifying redirect behavior, increasing max redirects, disabling redirects, and implementing custom redirect handling. It also offers best practices for handling redirects and answers frequently asked questions about the error.

The Redirect Ninja's Guide to Mastering Python Requests

Author: Mohan Ganesan

Date: Oct 31, 2023

Learn how to handle redirects in web scraping using Python's Requests module. Master techniques like sessions, custom redirect handlers, and inspecting redirects.

Difference between urllib and urllib2

Author: Mohan Ganesan

Date: Feb 6, 2024

urllib for simple HTTP requests, urllib2 for robust HTTPS, redirects, custom headers, and error handling.

urllib attribute error

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in Python provides functions for fetching data from the web. Common errors include attribute errors and invalid URLs. Handling redirects and errors is important.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!