Stories from the Web Crawling trenches in Perl

Web Scraping with Perl & ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Web scraping in Perl with ChatGPT assistance. Use HTML::TreeBuilder and WWW::Mechanize for data extraction. Generate code snippets and explanations with ChatGPT.

How to Build a Super Simple HTTP Proxy in Perl in just 20 lines of code

Author: Mohan Ganesan

Date: Oct 1, 2023

Build a basic HTTP proxy server in Perl using less than 20 lines of code. Use rotating proxy service to avoid IP blocking.

Scraping All Images from a Website with Perl

Author: Mohan Ganesan

Date: Dec 13, 2023

Guide to scraping image URLs and data from a Wikipedia page using Perl script. Extracts names, groups, local names, and image URLs for dog breeds.

Downloading Images from a Website with Perl and Mojo::DOM

Author: Mohan Ganesan

Date: Oct 15, 2023

Learn how to use Perl and modules like LWP::UserAgent and Mojo::DOM to download images of dog breeds from a Wikipedia page.

Scrape Any Website with OpenAI Function Calling in Perl

Author: Mohan Ganesan

Date: Sep 25, 2023

Web scraping with OpenAI in Perl to extract product data from HTML using function calling.

Scraping Reddit Posts in Perl

Author: Mohan Ganesan

Date: Jan 9, 2024

Scraping Reddit using Perl to extract information from posts by parsing HTML and using UserAgent for data extraction.

Scraping Multiple Pages in Perl with LWP::UserAgent and HTML::TreeBuilder

Author: Mohan Ganesan

Date: Oct 15, 2023

Web scraping in Perl using LWP::UserAgent and HTML::TreeBuilder modules to extract data from multiple pages. Use XPath queries and proxies for efficient data extraction.

Building a Simple Proxy Rotator with Perl and Mojo

Author: Mohan Ganesan

Date: Oct 2, 2023

Use Mojo::UserAgent to fetch and parse proxy lists, extract proxies, refresh periodically, select a random proxy, and make proxied requests with LWP::UserAgent. Consider using a rotating proxy service like Proxies API to solve IP blocking problems.

Scraping Craigslist Listings with Perl

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Perl and modules LWP::UserAgent and HTML::TreeBuilder. Avoid IP blocking with a rotating proxy server.

Scraping Data from Wikipedia with Perl

Author: Mohan Ganesan

Date: Dec 6, 2023

Scraping tabular data from Wikipedia using Perl. Extract and utilize structured data from Wikipedia pages.

Scraping New York Times News Headlines in Perl

Author: Mohan Ganesan

Date: Dec 6, 2023

Learn how to scrape the NYT website using Perl, LWP::UserAgent, and Mojo::DOM. Extract headlines and links programmatically.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!