Stories from the Web Crawling trenches in broken links

How To Find All URLs On A Domain's Website

Author: Mohan Ganesan

Date: Apr 30, 2024

Find all URLs on a website for SEO analysis, broken link detection, competitive analysis, web scraping, and website migration.

urllib read

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides functionality for retrieving data from URLs. It allows you to fetch web pages, decode and parse HTML, and handle errors. Practical examples include web scraping and checking broken links.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!