Scraping Multiple Pages in Ruby with Nokogiri

Oct 15, 2023 · 3 min read

Web scraping is useful to programmatically extract data from websites. Often you need to scrape multiple pages from a site to gather complete information. In this article, we'll see how to scrape multiple pages in Ruby using the Nokogiri library.


To follow along, you'll need:

  • Basic Ruby knowledge
  • Ruby installed
  • Nokogiri gem installed:
  • gem install nokogiri

    Import Gems

    We'll need the following gems:

    require 'nokogiri'
    require 'open-uri'

    Define Base URL

    We'll scrape a blog - The page URLs follow a pattern:


    Let's define the base URL pattern:

    base_url = '<>'

    The %d allows us to insert the page number.

    Specify Number of Pages

    Next, we'll specify how many pages to scrape. Let's scrape the first 5 pages:

    num_pages = 5

    Loop Through Pages

    We can now loop from 1 to num_pages and construct the URL for each page:

    (1..num_pages).each do |page_num|
      # Construct page URL
      url = base_url % page_num
      # Code to scrape each page

    Send Request and Parse HTML

    Inside the loop, we'll open the page URL and parse the HTML using Nokogiri:

    html = open(url)
    doc = Nokogiri::HTML(html)

    This gives us a parsed HTML document to extract data from.

    Extract Data

    Now within the loop we can use doc to find and extract data from each page.

    For example, to get all article elements:

    articles = doc.css('article')

    We can loop through articles and extract information like title, URL, author etc.

    Full Code

    Our full code to scrape 5 pages is:

    require 'nokogiri'
    require 'open-uri'
    base_url = '<>'
    num_pages = 5
    (1..num_pages).each do |page_num|
      url = base_url % page_num
      html = open(url)
      doc = Nokogiri::HTML(html)
      articles = doc.css('article')
      articles.each do |article|
        # Extract data from article
        title = article.at_css('h2.entry-title').text
        url = article.at_css('a.entry-title-link')['href']
        author = article.at_css(' a').text
        categories = article.css('div.entry-categories a').map(&:text)
        # Print extracted data
        puts "Title: #{title}"
        puts "URL: #{url}"
        puts "Author: #{author}"
        puts "Categories: #{categories.join(', ')}"

    This allows us to scrape and extract data from multiple pages sequentially. The code can be extended to scrape any number of pages.


  • Use a base URL pattern with %d placeholder
  • Loop through pages with Range#each
  • Construct each page URL
  • Open page and parse HTML with Nokogiri
  • Find and extract data inside loop
  • Print or store scraped data
  • Web scraping enables collecting large datasets programmatically. With the techniques here, you can scrape and extract information from multiple pages of a website in Ruby.

    While these examples are great for learning, scraping production-level sites can pose challenges like CAPTCHAs, IP blocks, and bot detection. Rotating proxies and automated CAPTCHA solving can help.

    Proxies API offers a simple API for rendering pages with built-in proxy rotation, CAPTCHA solving, and evasion of IP blocks. You can fetch rendered pages in any language without configuring browsers or proxies yourself.

    This allows scraping at scale without headaches of IP blocks. Proxies API has a free tier to get started. Check out the API and sign up for an API key to supercharge your web scraping.

    With the power of Proxies API combined with Python libraries like Beautiful Soup, you can scrape data at scale without getting blocked.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: