Scraping eBay Listings with Kotlin and HttpClient in 2023

Oct 5, 2023 ยท 3 min read

eBay is one of the largest online marketplaces with millions of active listings at any given time. In this tutorial, we'll walk through how to scrape and extract key data from eBay listings using Kotlin and the HttpClient library.

Setup

We'll need to add HttpClient to our dependencies:

implementation("io.ktor:ktor-client-cio:2.0.0")

And import it:

import io.ktor.client.*

We'll also define the eBay URL and a header for the user agent:

val url = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"

val userAgent = "Mozilla/5.0..."

Replace the user agent string with your browser's user agent.

Fetch the Listings Page

We can make the GET request using HttpClient:

val client = HttpClient()

val response: HttpResponse = client.get(url) {
    header("User-Agent", userAgent)
}

val html = response.readText()

The user agent header is added to the request.

Extract Listing Data

To parse the HTML, we can use Jsoup:

import org.jsoup.Jsoup

val doc = Jsoup.parse(html)

val listings = doc.select("div.s-item__info")

for (listing in listings) {

    val title = listing.select("div.s-item__title").text()
    val url = listing.select("a.s-item__link").attr("href")
    val price = listing.select("span.s-item__price").text()

    // Extract other fields like seller, shipping, etc

}

We select elements by CSS selector and extract text/attributes.

Print Results

We can print the extracted data:

print("Title: $title")
print("URL: $url")
print("Price: $price")

println("=".repeat(50)) // Separator between listings

This will output each listing's info.

Full Code

Here is the full code to scrape eBay listings:

import io.ktor.client.*
import org.jsoup.Jsoup

val url = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"
val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

val client = HttpClient()

val response: HttpResponse = client.get(url) {
  header("User-Agent", userAgent)
}

val html = response.readText()

val doc = Jsoup.parse(html)

val listings = doc.select("div.s-item__info")

for (listing in listings) {

  val title = listing.select("div.s-item__title").text()
  val url = listing.select("a.s-item__link").attr("href")
  val price = listing.select("span.s-item__price").text()

  val details = listing.select("div.s-item__subtitle").text()
  val sellerInfo = listing.select("span.s-item__seller-info-text").text()
  val shippingCost = listing.select("span.s-item__shipping").text()
  val location = listing.select("span.s-item__location").text()
  val sold = listing.select("span.s-item__quantity-sold").text()

  print("Title: $title")
  print("URL: $url")
  print("Price: $price")
  print("Details: $details")
  print("Seller: $sellerInfo")
  print("Shipping: $shippingCost")
  print("Location: $location")
  print("Sold: $sold")

  println("=".repeat(50))

}

Browse by tags:

Browse by language:

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!