Stories from the Web Crawling trenches in Puppeteer

The Complete Puppeteer Cheatsheet

Author: Mohan Ganesan

Date: Dec 6, 2023

Puppeteer is a Node.js library for automating UI testing, scraping, and screenshot testing using headless Chrome.

How to Use Proxies with Puppeteer in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to effectively use proxies with Puppeteer for web scraping, including the importance of proxies, configuring proxies in Puppeteer, rotating multiple proxies to avoid blocks, configuring authentication for premium proxies, and advanced proxy chaining. Discover common issues and troubleshooting tips, as well as criteria for selecting proxy services. Consider leveraging Proxies API for uninterrupted web scraping with worldwide locations, built-in rotation, JavaScript rendering, CAPTCHA solving, and high availability.

Capturing Screenshots with Puppeteer - An advanced guide

Author: Mohan Ganesan

Date: Jan 9, 2024

Puppeteer is a Node.js library for controlling headless Chrome, ideal for web scraping and automation tasks. It allows you to automate browser actions, capture screenshots, and perform advanced tasks like emulating mobile devices and simulating network conditions.

Bypassing CAPTCHAs with Puppeteer

Author: Mohan Ganesan

Date: Oct 4, 2023

Automate captcha solving using Puppeteer and headless Chrome with the help of a captcha solving service like 2Captcha.

Building a Simple Proxy Rotator with JavaScript and Puppeteer

Author: Mohan Ganesan

Date: Oct 2, 2023

Fetch and parse proxies using Puppeteer and cheerio, and select a random proxy for JavaScript projects.

How to Build a Super Simple HTTP proxy in Go in just 20 lines of code

Author: Mohan Ganesan

Date: Oct 1, 2023

Go is a great language for writing simple and efficient network applications. Learn how to build a basic HTTP proxy in Go in under 20 lines of code. To handle IP blocking, consider using a rotating proxy service like Proxies API.

Playwright vs Puppeteer: A Side-by-Side Comparison for Test Automation

Author: Mohan Ganesan

Date: Jan 9, 2024

Playwright and Puppeteer are popular browser testing tools that offer speed, capabilities, and reliability. Playwright has an advantage in terms of speed, browser support, and API design. Both tools are suitable for web app testing, but Puppeteer is recommended for web scraping tasks.

Puppeteer vs Selenium: A Web Scraper's Experience-Driven Comparison

Author: Mohan Ganesan

Date: Jan 9, 2024

Puppeteer and Selenium differ in their origins and purposes. Puppeteer is for web data extraction, while Selenium is for web app testing. When scraping data, Puppeteer requires explicit waits and explicit element lookup, while Selenium allows for configurable implicit waits and implicit element lookup. Both tools have their strengths and should be used accordingly.

Playwright vs Puppeteer for Web Scraping: How To Choose For Robust Data Extraction

Author: Mohan Ganesan

Date: Jan 9, 2024

Playwright and Puppeteer are both powerful tools for web scraping, but Puppeteer has an edge in speed and stealth capabilities, while Playwright excels in handling complex page state changes and offers a more flexible data extraction. Both libraries can serve most scraping needs, but Puppeteer is the top choice for advanced scenarios.

What are the limitations of BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a Python library for parsing and extracting data from HTML and XML documents. It struggles with modern JavaScript sites and cannot bypass most bot protections. CSS selectors and navigation logic can get complex. Consider alternatives like Scrapy, Puppeteer, or Playwright for professional web scraping.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!