Stories from the Web Crawling trenches in credentials

Web Scraping Websites with Login Example Using Python

Author: Mohan Ganesan

Date: Oct 4, 2023

Analyze login form, craft payload, post login request, use session to stay logged in, hide credentials, scrape data from restricted pages!

HttpWebRequest Proxies in C# in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

The article explains how to direct HttpWebRequest traffic through a proxy using the WebProxy class. It covers creating a WebProxy, assigning it to HttpWebRequest, proxy authentication, default system proxy settings, and making requests via proxy.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!