Stories from the Web Crawling trenches in CSS selectors

The Complete BeautifulSoup Cheatsheet with Examples

Author: Mohan Ganesan

Date: Oct 4, 2023

This cheatsheet covers the full BeautifulSoup 4 API with practical examples. It provides a comprehensive guide to web scraping and HTML parsing using Python's BeautifulSoup library.

The Ultimate Floki Cheatsheet for Elixir

Author: Mohan Ganesan

Date: Oct 31, 2023

Floki makes it easy to parse and query HTML documents in Elixir using CSS selectors and tree traversal.

The Ultimate Jsoup Cheatsheet in Java

Author: Mohan Ganesan

Date: Oct 31, 2023

Jsoup is a Java library for parsing and manipulating HTML using DOM, CSS, and jquery-like methods.

The Ultimate Rvest Cheatsheet in R

Author: Mohan Ganesan

Date: Oct 31, 2023

rvest is a package in R for web scraping and data extraction from HTML using CSS selectors. It also provides functions for parsing and navigating HTML documents. Additional features include handling issues, advanced usage with RSelenium, best practices, troubleshooting, and tips and tricks. The package is useful for scraping websites ethically and efficiently, processing extracted data, and handling large datasets.

The Ultimate JSoup Kotlin Cheatsheet

Author: Mohan Ganesan

Date: Oct 31, 2023

JSoup is a Java library for working with real-world HTML. It provides a convenient API for extracting and manipulating data from HTML documents.

Web Scraping in CSharp - The Ultimate Guide

Author: Mohan Ganesan

Date: Mar 24, 2024

Learn web scraping with C# using powerful libraries like HtmlAgilityPack and AngleSharp. Understand the importance of XPath and CSS selectors for extracting data from HTML. Overcome challenges like dynamic content and anti-scraping measures. Rotate user agents and headers to mimic human behavior and avoid detection.

CSS Selectors vs XPath with BeautifulSoup: How to Choose the Right Selector

Author: Mohan Ganesan

Date: Oct 6, 2023

CSS selectors and XPath expressions are powerful techniques for parsing and extracting data from HTML and XML. CSS selectors offer simplicity and readability, while XPath provides unmatched query power and flexibility. Combining both can give you a robust toolkit for efficient data extraction.

A Comprehensive Guide to Searching with CSS Selectors and Attributes in BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

The BeautifulSoup library provides powerful techniques for searching and extracting data from HTML and XML documents using CSS selectors. Mastering these techniques will enhance web scraping and parsing capabilities.

The Ultimate JSoup Scala Cheatsheet

Author: Mohan Ganesan

Date: Oct 31, 2023

JSoup is a Java library for working with real-world HTML. It provides a convenient API for extracting and manipulating data from HTML documents.

Scraping All Images from a Website with Java

Author: Mohan Ganesan

Date: Dec 13, 2023

Web scraping is the process of extracting data from websites automatically. This article explains how to scrape dog breed images from a Wikipedia page using Java and Jsoup library. It also discusses the use of CSS selectors and overcoming IP blocking.

What are the limitations of BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a Python library for parsing and extracting data from HTML and XML documents. It struggles with modern JavaScript sites and cannot bypass most bot protections. CSS selectors and navigation logic can get complex. Consider alternatives like Scrapy, Puppeteer, or Playwright for professional web scraping.

A Guide to BeautifulSoup's CSS Selector Capabilities

Author: Mohan Ganesan

Date: Oct 6, 2023

The BeautifulSoup library supports searching and extracting elements from HTML and XML documents using CSS selectors, making it a powerful tool for web scraping.

Scraping Reddit Posts with PHP

Author: Mohan Ganesan

Date: Jan 9, 2024

Web scraping with PHP to extract data from Reddit using DOM parsing, CSS selectors, and cURL.

Scraping New York Times News Headlines with Rust

Author: Mohan Ganesan

Date: Dec 6, 2023

Automatically collect and analyze data from websites using web scraping in Rust. Learn how to make structured requests, parse HTML, and use CSS selectors to extract information.

Web Scraping New York Times News Headlines in Ruby

Author: Mohan Ganesan

Date: Dec 6, 2023

Web scraping allows automatic data extraction from websites. This article demonstrates web scraping using Ruby, Nokogiri, and Net::HTTP. It covers CSS selectors, handling errors, and overcoming IP blocks.

Scraping Real Estate Listings From Realtor in Java

Author: Mohan Ganesan

Date: Jan 9, 2024

Scrape real estate listing data from Realtor.com using Jsoup, a Java library. Analyze trends or build applications with large-scale housing data.

Scraping Real Estate Listings From Realtor with Go

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to scrape real estate listing data from Realtor.com using Go and the goquery library. Use web scraping to collect and analyze housing data.

Scraping Reddit Posts with R

Author: Mohan Ganesan

Date: Jan 9, 2024

Scrape data from Reddit posts using R code, handling responses, extracting information, and iterating through multiple posts.

Scraping Real Estate Listings From Realtor in Kotlin

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to use Jsoup for web scraping to extract key details from real estate listings on Realtor.com. This comprehensive guide covers crafting GET requests, selecting HTML elements with CSS selectors, extracting and transforming text, and dealing with missing data. By the end, you'll be able to scrape details like broker name, status, price, beds, baths, square footage, lot size, and full address from any Realtor.com search page.

What are the features of BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

Web scraping with BeautifulSoup: a powerful Python library for extracting data from websites using simple API and CSS selectors.

Introduction to Scraping Reddit Posts in Scala

Author: Mohan Ganesan

Date: Jan 9, 2024

Beginner-friendly guide to scrape content from Reddit using Scala and Play Framework's WS library. Extract key information like post titles, permalinks, authors, and scores from Reddit posts on a webpage.

Scraping Reddit Posts in Elixir

Author: Mohan Ganesan

Date: Jan 9, 2024

Web scraping tutorial using Elixir code to extract post information from Reddit. Learn how to install dependencies, make requests, parse HTML, and use CSS selectors.

Scraping Real Estate Listings From Realtor in Scala

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to extract key details from real estate listings on Realtor.com using Jsoup, a Java library for web scraping.

Scraping Yelp Business Listings Using Perl

Author: Mohan Ganesan

Date: Dec 6, 2023

Web scraping is the process of extracting data from websites through automated scripts. This article provides a beginner tutorial on scraping business listings from Yelp using modules like HTML::TreeBuilder and LWP::UserAgent.

Is BeautifulSoup easy to learn?

Author: Mohan Ganesan

Date: Feb 5, 2024

Web scraping with BeautifulSoup is a valuable skill for data scientists and Python developers. It's beginner-friendly and has convenient methods for extracting data. Learning CSS selectors is necessary for effective use.

Do I need to learn HTML for web scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

HTML knowledge is useful but not necessary for web scraping. Tools like BeautifulSoup and selector gadgets can be used to extract data without deep HTML knowledge.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!