Stories from the Web Crawling trenches in data scraping

How to Tell if a Website is Scrapable

Author: Mohan Ganesan

Date: Feb 20, 2024

Determine if a website can be scraped by checking the robots.txt file, analyzing the page source, checking for CAPTCHAs, and testing scraping a page.

Web Scraping Websites with Login Example Using Python

Author: Mohan Ganesan

Date: Oct 4, 2023

Analyze login form, craft payload, post login request, use session to stay logged in, hide credentials, scrape data from restricted pages!

Web Scraping into Excel using ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Web scraping with ChatGPT allows easy extraction of data from websites and saving it in Excel using Python code. Use Pandas to format and output data. Get started now!

Does YouTube allow scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

YouTube restricts data scraping to protect its platform and users. Exceptions include limited personal use and research purposes.

What is Data Scraping? Techniques and Top 6 Tools

Author: Mohan Ganesan

Date: Apr 30, 2024

Data scraping is the process of extracting data from websites or other sources. It involves automating the collection of structured data from various online platforms.

Scraping LinkedIn Data: What's Allowed and Best Practices

Author: Mohan Ganesan

Date: Feb 20, 2024

LinkedIn is a popular social media platform with over 800 million members. While data scraping is prohibited, individuals can manually access and collect public information in a responsible way.

What is the difference between web scraping and data scraping?

Author: Mohan Ganesan

Date: Feb 5, 2024

Web scraping extracts data from web pages, while data scraping is a broader term for extracting data from any online source.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!