Stories from the Web Crawling trenches in SEO

How To Find All URLs On A Domain's Website

Author: Mohan Ganesan

Date: Apr 30, 2024

Find all URLs on a website for SEO analysis, broken link detection, competitive analysis, web scraping, and website migration.

The Role of Web Scraping in SEO

Author: Mohan Ganesan

Date: Feb 22, 2024

Web scraping is a useful technique in SEO for competitor research, backlink analysis, rank tracking, and content gap analysis.

How Google Leverages Data Collection Methods Like Web Scraping

Author: Mohan Ganesan

Date: Feb 20, 2024

Google relies on web scraping for data collection, SEO, AI models, Knowledge Graph, and local business info. However, it raises ethical concerns.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!