Downloading Images from a Website with Kotlin and Jsoup

Oct 15, 2023 · 4 min read

In this article, we will learn how to use Kotlin and the Jsoup library to download all the images from a Wikipedia page.

—-

Overview

The goal is to extract the names, breed groups, local names, and image URLs for all dog breeds listed on this Wikipedia page. We will store the image URLs, download the images and save them to a local folder.

Here are the key steps we will cover:

  1. Import required packages
  2. Send HTTP request to fetch the Wikipedia page
  3. Parse the page HTML using Jsoup
  4. Find the table with dog breed data using a CSS selector
  5. Iterate through the table rows
  6. Extract data from each column
  7. Download images and save locally
  8. Print/process extracted data

Let's go through each of these steps in detail.

Imports

We need these packages:

import java.net.HttpURLConnection
import org.jsoup.Jsoup
import org.jsoup.nodes.Document
  • HttpURLConnection - Sends HTTP requests
  • Jsoup - Parses HTML/XML
  • Send HTTP Request

    To download the web page:

    val url = "<https://commons.wikimedia.org/wiki/List_of_dog_breeds>"
    val connection = URL(url).openConnection() as HttpURLConnection
    connection.setRequestProperty("User-Agent", "KotlinScraper")
    
    val document = connection.inputStream.bufferedReader().use{ it.readText() }
    

    We open a connection and provide a custom user-agent header.

    Parse HTML

    To parse the HTML:

    val html: Document = Jsoup.parse(document)
    

    The Jsoup Document represents parsed HTML.

    Find Breed Table

    We use a CSS selector to find the table element:

    val table = html.select("table.wikitable.sortable")
    

    This selects the

    tag with the required CSS classes.

    Iterate Through Rows

    We can iterate through the rows like this:

    for (row in table.select("tr")) {
    
      // Extract data
    
    }
    

    We loop through each

    element within the table.

    Extract Column Data

    Inside the loop, we extract the column data:

    val cells = row.select("td, th")
    
    val name = cells[0].select("a").text()
    val group = cells[1].text()
    
    val localName = cells[2].select("span").text() ?: ""
    
    val img = cells[3].select("img").attr("src")
    

    We use text() for text and attr() for attributes.

    Download Images

    To download and save images:

    if (img != "") {
    
      val imageStream = URL(img).openStream()
    
      Files.copy(imageStream, Paths.get("dog_images/$name.jpg"))
    
    }
    

    We open the image stream and save it to a file.

    Store Extracted Data

    We can store the extracted data in lists:

    names.add(name)
    groups.add(group)
    localNames.add(localName)
    images.add(img)
    

    The lists can then be processed as needed.

    And that's it! Here is the full code:

    // Imports
    import java.net.HttpURLConnection
    import org.jsoup.Jsoup
    import org.jsoup.nodes.Document
    import java.io.File
    import java.nio.file.Files
    import java.nio.file.Paths
    
    // Lists to store data
    val names = mutableListOf<String>()
    val groups = mutableListOf<String>()
    val localNames = mutableListOf<String>()
    val images = mutableListOf<String>()
    
    // Fetch HTML
    val url = "<https://commons.wikimedia.org/wiki/List_of_dog_breeds>"
    val connection = URL(url).openConnection() as HttpURLConnection
    connection.setRequestProperty("User-Agent", "KotlinScraper")
    
    val document = connection.inputStream.bufferedReader().use{ it.readText() }
    
    // Parse HTML
    val html: Document = Jsoup.parse(document)
    
    // Find table
    val table = html.select("table.wikitable.sortable")
    
    // Iterate rows
    for (row in table.select("tr")) {
    
      // Get cells
      val cells = row.select("td, th")
    
      // Extract data
      val name = cells[0].select("a").text()
      val group = cells[1].text()
    
      val localName = cells[2].select("span").text() ?: ""
    
      val img = cells[3].select("img").attr("src")
    
      // Download image
      if (img != "") {
    
        val imageStream = URL(img).openStream()
    
        Files.copy(imageStream, Paths.get("dog_images/$name.jpg"))
    
      }
    
      // Store data
      names.add(name)
      groups.add(group)
      localNames.add(localName)
      images.add(img)
    
    }
    

    This provides a complete Kotlin solution using Jsoup to scrape data and images from HTML tables. The same approach can apply to many websites.

    While these examples are great for learning, scraping production-level sites can pose challenges like CAPTCHAs, IP blocks, and bot detection. Rotating proxies and automated CAPTCHA solving can help.

    Proxies API offers a simple API for rendering pages with built-in proxy rotation, CAPTCHA solving, and evasion of IP blocks. You can fetch rendered pages in any language without configuring browsers or proxies yourself.

    This allows scraping at scale without headaches of IP blocks. Proxies API has a free tier to get started. Check out the API and sign up for an API key to supercharge your web scraping.

    With the power of Proxies API combined with Python libraries like Beautiful Soup, you can scrape data at scale without getting blocked.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!