eBay is one of the largest online marketplaces with millions of active listings at any given time. In this tutorial, we'll walk through how to scrape and extract key data from eBay listings using Kotlin and the HttpClient library.
Setup
We'll need to add HttpClient to our dependencies:
implementation("io.ktor:ktor-client-cio:2.0.0")
And import it:
import io.ktor.client.*
We'll also define the eBay URL and a header for the user agent:
val url = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"
val userAgent = "Mozilla/5.0..."
Replace the user agent string with your browser's user agent.
Fetch the Listings Page
We can make the GET request using HttpClient:
val client = HttpClient()
val response: HttpResponse = client.get(url) {
header("User-Agent", userAgent)
}
val html = response.readText()
The user agent header is added to the request.
Extract Listing Data
To parse the HTML, we can use Jsoup:
import org.jsoup.Jsoup
val doc = Jsoup.parse(html)
val listings = doc.select("div.s-item__info")
for (listing in listings) {
val title = listing.select("div.s-item__title").text()
val url = listing.select("a.s-item__link").attr("href")
val price = listing.select("span.s-item__price").text()
// Extract other fields like seller, shipping, etc
}
We select elements by CSS selector and extract text/attributes.
Print Results
We can print the extracted data:
print("Title: $title")
print("URL: $url")
print("Price: $price")
println("=".repeat(50)) // Separator between listings
This will output each listing's info.
Full Code
Here is the full code to scrape eBay listings:
import io.ktor.client.*
import org.jsoup.Jsoup
val url = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"
val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
val client = HttpClient()
val response: HttpResponse = client.get(url) {
header("User-Agent", userAgent)
}
val html = response.readText()
val doc = Jsoup.parse(html)
val listings = doc.select("div.s-item__info")
for (listing in listings) {
val title = listing.select("div.s-item__title").text()
val url = listing.select("a.s-item__link").attr("href")
val price = listing.select("span.s-item__price").text()
val details = listing.select("div.s-item__subtitle").text()
val sellerInfo = listing.select("span.s-item__seller-info-text").text()
val shippingCost = listing.select("span.s-item__shipping").text()
val location = listing.select("span.s-item__location").text()
val sold = listing.select("span.s-item__quantity-sold").text()
print("Title: $title")
print("URL: $url")
print("Price: $price")
print("Details: $details")
print("Seller: $sellerInfo")
print("Shipping: $shippingCost")
print("Location: $location")
print("Sold: $sold")
println("=".repeat(50))
}