How To Use BeautifulSoup's find_all() Method

Oct 6, 2023 ยท 2 min read

The find_all() method in BeautifulSoup is used to find all tags or strings matching a given criteria in an HTML/XML document. It's a very useful method for scraping and parsing, but there are some key things to understand when using it effectively.

Returns a List

The find_all() method returns a list of all matching tags and strings. Even if there is only one match, it will be returned in a list.

soup.find_all('p') # Returns list of <p> tags

So you often need to loop over the result or index into it to get the first matching element.

Match by String, Regex, or Function

find_all() can search by:

  • A string matching the tag name
  • A regex pattern matching the tag name
  • A function that returns True if the tag matches certain criteria
  • For example:

    # String match
    # Regex match
    import re
    # Function match
    def has_class_name(tag):
      return tag.has_attr('class')

    Search Within a Tag

    Pass a tag as the first argument to find_all() to search within that part of the document only.

    content = soup.find(id="content")
    content.find_all('p') # Finds <p> tags inside div#content only

    Keyword Arguments

    find_all() accepts keyword arguments to filter matches by attribute values, like id or class.

    For example to find links:

    soup.find_all('a', href=True)

    This can make searching more precise.

    text Keyword

    A special keyword argument is text to search for strings instead of tags:

    soup.find_all(text="Hello World") # Finds text nodes


    Mastering find_all() is key to effective web scraping with BeautifulSoup. Understanding how to craft targeted searches makes extracting data much easier.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: