Stories from the Web Crawling trenches in test automation

Puppeteer vs Selenium: A Web Scraper's Experience-Driven Comparison

Author: Mohan Ganesan

Date: Jan 9, 2024

Puppeteer and Selenium differ in their origins and purposes. Puppeteer is for web data extraction, while Selenium is for web app testing. When scraping data, Puppeteer requires explicit waits and explicit element lookup, while Selenium allows for configurable implicit waits and implicit element lookup. Both tools have their strengths and should be used accordingly.

Mastering XPath Locators for Reliable Selenium Tests

Author: Mohan Ganesan

Date: Jan 9, 2024

Locators in test automation allow for the identification of elements on a web page. XPath locators are robust and flexible, making them ideal for scalable test automation. By mastering XPath syntax and operators, test engineers can construct dynamic locators to handle complex scenarios. Integrating XPath locators into Selenium scripts requires understanding the difference between finding a single element and multiple elements. Best practices include reusing locators through the Page Object Model pattern and handling exceptions carefully. Troubleshooting XPath issues involves verifying locator accuracy, outputting attribute values, and using more resilient variations. Overall, mastering XPath locators is crucial for successful UI test automation using Selenium.

Should I learn Selenium or Scrapy?

Author: Mohan Ganesan

Date: Feb 5, 2024

Automating tests with Selenium saves time and reduces errors, while Scrapy is better for large scale web scraping.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!