Stories from the Web Crawling trenches in XML

The Ultimate Nokogiri Cheat Sheet for Ruby

Author: Mohan Ganesan

Date: Oct 31, 2023

Nokogiri is a powerful HTML/XML parsing and scraping library for Ruby. This cheat sheet covers its extensive capabilities.

Introduction to Web Scraping with BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

Web scraping is the process of extracting data from websites through an automated procedure. Beautiful Soup is a Python library designed specifically for web scraping purposes. It provides parsing and navigation tools for extracting data from HTML and XML documents.

A Guide to Using XPath with BeautifulSoup for Powerful Web Scraping

Author: Mohan Ganesan

Date: Oct 6, 2023

XPath is a powerful querying language for selecting elements in XML and HTML documents, making web scraping with BeautifulSoup more robust and flexible.

The Ultimate HTML::TreeBuilder Cheatsheet in Perl

Author: Mohan Ganesan

Date: Oct 31, 2023

HTML::TreeBuilder is a Perl module for parsing and manipulating HTML and XML documents into a tree structure.

Finding Headers in BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

When parsing HTML and XML documents, accessing and working with headers is a common task. Understanding header tags in BeautifulSoup is important for efficient parsing and processing of documents.

The Complete Python HTML Parser Cheatsheet

Author: Mohan Ganesan

Date: Jan 9, 2024

The Python HTML parser allows you to parse HTML and XML documents and extract data. This article provides a comprehensive guide on how to use the parser effectively.

The Ultimate DOMDocument Cheat Sheet for PHP

Author: Mohan Ganesan

Date: Oct 31, 2023

DOMDocument allows manipulating HTML/XML documents in PHP. This cheat sheet is a comprehensive reference for working with DOMDocument.

A Comprehensive Guide to Searching with CSS Selectors and Attributes in BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

The BeautifulSoup library provides powerful techniques for searching and extracting data from HTML and XML documents using CSS selectors. Mastering these techniques will enhance web scraping and parsing capabilities.

How to Add Comments in JSON

Author: Mohan Ganesan

Date: Oct 4, 2023

JSON is a lightweight data format without native comment support. Use YAML or XML for commenting. JSONC is an emerging standard for comments in JSON.

What is the difference between Python ElementTree and BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

ElementTree is best for working with valid XML documents, while BeautifulSoup is designed for parsing potentially malformed real-world HTML.

The Ultimate NSXMLParser Cheatsheet

Author: Mohan Ganesan

Date: Oct 31, 2023

NSXMLParser allows parsing XML documents in Objective-C. It provides SAX style event-driven parsing.

CSS Selectors vs XPath with BeautifulSoup: How to Choose the Right Selector

Author: Mohan Ganesan

Date: Oct 6, 2023

CSS selectors and XPath expressions are powerful techniques for parsing and extracting data from HTML and XML. CSS selectors offer simplicity and readability, while XPath provides unmatched query power and flexibility. Combining both can give you a robust toolkit for efficient data extraction.

Find the text of the given tag using BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

The get_text() method in Python BeautifulSoup library is useful for extracting text from HTML and XML documents. It strips HTML tags, handles whitespace and nested tags, and ignores invisible text.

A Guide to BeautifulSoup's CSS Selector Capabilities

Author: Mohan Ganesan

Date: Oct 6, 2023

The BeautifulSoup library supports searching and extracting elements from HTML and XML documents using CSS selectors, making it a powerful tool for web scraping.

Web Scraping with Visual Basic & ChatGPT

Author: Mohan Ganesan

Date: Sep 25, 2023

Visual Basic provides a straightforward way to build web scrapers. ChatGPT is an AI assistant that can explain concepts and generate VB code for scraping.

Can BeautifulSoup use XPath?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup and XPath can complement each other to create powerful web scrapers, but be mindful of the performance tradeoff.

Scraping Craigslist Listings with Scala

Author: Mohan Ganesan

Date: Oct 1, 2023

Learn how to scrape Craigslist apartment listings using Scala and the play-ws library. Use XML parsing and a rotating proxy server to avoid IP blocking.

Difference Between find() and find_all() in BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

The find() and find_all() methods in Python BeautifulSoup library are used for searching and extracting elements from HTML and XML documents. find() returns the first matching element, while find_all() returns a list of all matching elements.

What are the limitations of BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a Python library for parsing and extracting data from HTML and XML documents. It struggles with modern JavaScript sites and cannot bypass most bot protections. CSS selectors and navigation logic can get complex. Consider alternatives like Scrapy, Puppeteer, or Playwright for professional web scraping.

Using BeautifulSoup and Requests for Powerful Web Scraping

Author: Mohan Ganesan

Date: Oct 6, 2023

Requests and BeautifulSoup are two Python libraries that complement each other beautifully for web scraping purposes. They provide a powerful toolkit for extracting data from websites.

Can BeautifulSoup parse XML?

Author: Mohan Ganesan

Date: Feb 5, 2024

Beautiful Soup is a Python library for parsing HTML and XML documents. It can parse XML documents with some limitations. For more advanced XML capabilities, consider using Python's built-in XML libraries or third-party libraries like lxml.

Is BeautifulSoup free?

Author: Mohan Ganesan

Date: Feb 5, 2024

Beautiful Soup is a free and open source Python library used for web scraping. It can handle messy HTML, easily find elements, and extract data. Install it using pip and add it to your developer toolkit!

Why is it called BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a popular Python library for web scraping and parsing HTML and XML documents, bringing structure to messy markup.

Is Scrapy faster than BeautifulSoup?

Author: Mohan Ganesan

Date: Feb 5, 2024

Scrapy is a faster dedicated web scraping framework while BeautifulSoup excels at parsing HTML/XML.

Is BeautifulSoup lxml or HTML?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a popular Python library for parsing HTML and XML documents. It doesn't parse documents itself, but uses other parsers like lxml and html.parser. It provides methods for navigating, searching, and modifying parsed document trees.

Parsing XML with BeautifulSoup

Author: Mohan Ganesan

Date: Oct 6, 2023

BeautifulSoup can parse and extract data from XML and HTML documents, making it useful for scraping and analyzing data. It can navigate and search the parsed tree, modify the tree, and output the modified XML. It can also convert a BeautifulSoup XML object back into a string and perform additional processing. Examples demonstrate parsing XML files, displaying extracted data in tables using Pandas, and saving extracted data to CSV files.

Is BeautifulSoup a library or module?

Author: Mohan Ganesan

Date: Feb 5, 2024

BeautifulSoup is a library in Python for parsing, navigating, and searching HTML and XML documents.

What is BeautifulSoup 4?

Author: Mohan Ganesan

Date: Feb 5, 2024

Web scraping is the process of extracting data from websites using Python's BeautifulSoup library, which provides methods to parse and search HTML and XML documents. It is popular due to its simplicity and extensive features.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!