Stories from the Web Crawling trenches in dynamic websites

Using Python Requests Module with Dropdown Options

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests module is a valuable tool for web scraping, especially when dealing with dropdown menus. This article demonstrates how to use Requests to interact with dropdowns and extract the necessary data.

How to SCRAPE DYNAMIC Websites with Selenium

Author: Mohan Ganesan

Date: Oct 4, 2023

Web scraping dynamic websites with Selenium for automation and data extraction. Consider using ProxiesAPI for robust and scalable commercial scraping projects.

Is BeautifulSoup a data analysis tool?

Author: Mohan Ganesan

Date: Feb 5, 2024

Python library BeautifulSoup enables data extraction and analysis from web pages. Integrating with Pandas allows for deeper analysis and tracking changes to sites over time.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!