What are the three basic parts of a scraper?

Feb 22, 2024 ยท 4 min read

The internet contains a treasure trove of useful information, but unfortunately that information is not always in a format that's easy for us to collect and analyze. This is where web scrapers come in handy.

Web scrapers allow you to programmatically extract data from websites, transform it into a structured format like a CSV or JSON file, and save it to your computer for further analysis. Whether you need to gather data for a research project, populate a database, or build a price comparison site, scrapers are an invaluable tool.

In this post, we'll explore the three essential parts that make up a web scraper: the downloader, the parser, and the data exporter. Understanding the role each component plays will give you the foundations to build your own scrapers or tweak existing ones for your specific needs.

The Scraper's Brain: The Downloader

The first order of business for any scraper is to download the HTML code of the target webpage. This raw HTML contains all the underlying data we want to extract.

The downloader handles connecting to the website and pulling down the HTML code. Some popular downloader libraries in Python include:

  • Requests - Provides simple methods to issue standard HTTP requests like GET and POST. Handles cookies, redirects etc.
  • Scrapy - A framework focused solely on web scraping. More complex but very powerful.
  • Selenium - Automates a real web browser like Chrome or Firefox. Needed for sites that require JavaScript rendering.
  • Here is some sample code using the Requests library to download the HTML of example.com:

    import requests
    url = 'http://example.com'
    response = requests.get(url)
    html = response.text

    The downloader gives the scraper access to render the HTML source code from practically any public website.

    The Parser: Extracting Data from HTML

    Armed with the raw HTML source, the scraper next needs to parse through it and extract the relevant data. This requires identifying patterns in the HTML and using the right parsing technique to isolate the data.

    Some common parsing approaches include:

  • Regular expressions - Powerful string matching patterns to extract text. Useful for finding phone numbers, email addresses etc buried in text.
  • CSS selectors - Query elements by CSS class/id names like jQuery. Great for scraping common site layouts.
  • XPath selectors - Traverse XML style hierarchy of HTML elements. Handy when CSS classes are unpredictable.
  • HTML parsing libraries - Python libs like Beautiful Soup and Lxml to programmatically navigate the DOM tree.
  • For example to extract all the links from a page, we first need to download the HTML, then parse it with Beautiful Soup, find all anchor tag elements, and extract the link URLs like so:

    import requests
    from bs4 import BeautifulSoup
    url = 'http://example.com'
    resp = requests.get(url)
    soup = BeautifulSoup(resp.text, 'html.parser')
    links = []
    for link in soup.find_all('a'):

    The parsing stage is where the scraper really has to get its hands dirty and scrape the data out of messy HTML. The right technique depends largely on the structure of the site you are scraping.

    Exporting & Storing: The Scraper's Treasure Chest

    The final piece of our web scraping puzzle is to store the extracted data. Often we want to export and save this data for future analysis.

    Some handy formats to store scraper output include:

  • CSV - Simple comma separated values file. Works well with tables.
  • JSON - Portable file format that plays nice with Javascript.
  • SQL database - Store data in relational database tables.
  • Excel sheet - Export scrape results straight into XLSX sheets.
  • We can use Python's CSV module to export the links from our example into a CSV file:

    import csv 
    with open('links.csv', 'w') as f:
        writer = csv.writer(f) 

    The data pipeline ends with the scraper output saved locally in an easy to parse format. This data can now be loaded and processed by other applications.

    Bringing It All Together

    While scrapers can get complex, every web scraper fundamentally performs these three steps:

    1. Download raw HTML from the site
    2. Parse and extract relevant data
    3. Export structured data for further use

    Understanding this anatomy equips you with the core concepts needed to build your own scrapers.

    Of course in practice there is still much to learn regarding handling JavaScript sites, dealing with pagination, avoiding bot detection systems, maximizing performance and more.

    But the techniques discussed here form the backbone of any scraper. Whether you want to gather data from the web for research, business intelligence or personal projects, scrapers are an essential tool to have in your toolkit.

    So sharpen those scrapers and happy harvesting! The vast bounty of the internet awaits.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: