Scraping Booking.com Property Listings in Go in 2023

Oct 15, 2023 · 4 min read

In this article, we will learn how to scrape property listings from Booking.com using Go. We will use the net/http and goquery libraries to fetch the HTML content and then extract key information like property name, location, ratings, etc.

Prerequisites

To follow along, you will need:

  • Go installed on your system
  • Basic knowledge of Go programming
  • golang.org/x/net/html for HTML parsing
  • Importing Packages

    At the top of your Go file, import the required packages:

    import (
      "net/http"
      "github.com/PuerkitoBio/goquery"
    )
    

    net/http provides HTTP client capabilities.

    goquery allows jQuery-style HTML parsing and traversal.

    Defining the Target URL

    Let's define the URL we want to scrape:

    url := "<https://www.booking.com/searchresults.html?ss=New+York&>..."
    

    We won't paste the full URL here.

    Setting a User Agent

    We need to set a valid user agent header:

    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...")
    

    This will make the request appear to come from a real browser.

    Fetching the HTML Page

    We can use the http Client to get the page HTML:

    resp, _ := client.Do(req)
    defer resp.Body.Close()
    
    if resp.StatusCode == 200 {
    
      // Parse HTML
    
    }
    

    We make sure the request succeeded before parsing.

    Parsing the HTML

    To parse the HTML, we use goquery's NewDocumentFromReader method:

    doc, _ := goquery.NewDocumentFromReader(resp.Body)
    

    This loads the HTML into a goquery Document.

    Extracting Property Cards

    The property cards have a data-testid attribute we can search for:

    doc.Find("div[data-testid='property-card']").Each(func(i int, card *goquery.Selection) {
    
      // Extract data from card
    
    })
    

    This finds all matching

    elements.

    Extracting Title

    To get the title, we search for the data-testid="title" element:

    title := card.Find("div[data-testid='title']").Text()
    

    We grab the text contents.

    Extracting Location

    Similarly, the address is under a data-testid="address" element:

    location := card.Find("span[data-testid='address']").Text()
    

    The pattern is the same for other fields.

    Extracting Rating

    The star rating aria-label contains the score:

    rating := card.Find("div.e4755bbd60").Attr("aria-label")
    

    Here we get the aria-label attribute from the div.

    Extracting Review Count

    The review count text is inside a class="abf093bdfe" element:

    reviewCount := card.Find("div.abf093bdfe").Text()
    

    Extracting Description

    The description is in a class="d7449d770c" element:

    description := card.Find("div.d7449d770c").Text()
    

    Printing the Data

    We can print out the extracted data:

    fmt.Println("Name:", title)
    fmt.Println("Location:", location)
    fmt.Println("Rating:", rating)
    // etc...
    

    The full code for scraping each property card is available on GitHub.

    And that covers scraping Booking.com property listings in Go! Let me know if you have any other questions.

    Full Code

    Here is the complete Go code:

    package main
    
    import (
      "fmt"
      "net/http"
    
      "github.com/PuerkitoBio/goquery"
    )
    
    func main() {
    
      url := "https://www.booking.com/searchresults.en-gb.html?ss=New+York&checkin=2023-03-01&checkout=2023-03-05&group_adults=2"
    
      client := &http.Client{}
      req, _ := http.NewRequest("GET", url, nil)
      req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...")
    
      resp, _ := client.Do(req)
      defer resp.Body.Close()
    
      if resp.StatusCode == 200 {
    
        doc, _ := goquery.NewDocumentFromReader(resp.Body)
    
        doc.Find("div[data-testid='property-card']").Each(func(i int, card *goquery.Selection) {
    
          title := card.Find("div[data-testid='title']").Text()
          location := card.Find("span[data-testid='address']").Text()
          rating := card.Find("div.e4755bbd60").Attr("aria-label")
          reviewCount := card.Find("div.abf093bdfe").Text()
          description := card.Find("div.d7449d770c").Text()
    
          fmt.Println("Name:", title)
          fmt.Println("Location:", location)
          fmt.Println("Rating:", rating)
          fmt.Println("Review Count:", reviewCount)
          fmt.Println("Description:", description)
    
        })
    
      }
    
    }
    

    While these examples are great for learning, scraping production-level sites can pose challenges like CAPTCHAs, IP blocks, and bot detection. Rotating proxies and automated CAPTCHA solving can help.

    Proxies API offers a simple API for rendering pages with built-in proxy rotation, CAPTCHA solving, and evasion of IP blocks. You can fetch rendered pages in any language without configuring browsers or proxies yourself.

    This allows scraping at scale without headaches of IP blocks. Proxies API has a free tier to get started. Check out the API and sign up for an API key to supercharge your web scraping.

    With the power of Proxies API combined with Python libraries like Beautiful Soup, you can scrape data at scale without getting blocked.

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!