Stories from the Web Crawling trenches in login credentials

Dealing with 403 Forbidden Errors in BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

Ways to handle and bypass 403 Forbidden errors in web scraping: checking error codes, using user agents, authenticating with login credentials, waiting and retrying, using proxies.

Automate Website Logins with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Logging into websites made easy with Python's requests module. Replicate login process, handle response codes, automate workflows.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!