Automate Search Form Submission with Python Requests

Feb 3, 2024 ยท 2 min read

Submitting forms is a common task when scraping the web or automating workflows. Manually filling out forms over and over wastes time. Python requests allows you to easily submit forms programmatically.

Submitting Search Forms

Many websites have search forms to filter content. For example, an ecommerce site may have a search box to find products. Python requests can automatically submit these forms so you can retrieve the filtered results in your code.

Here is an example search form:

<form action="/search" method="get">
  <input type="text" name="query">
  <input type="submit" value="Search">
</form>

To submit this form in Python:

import requests

url = 'https://www.example.com/search'
payload = {'query': 'hello world'}

r = requests.get(url, params=payload)
print(r.url)
# https://www.example.com/search?query=hello+world

The key things to note:

  • Use the requests.get() method to submit GET forms
  • Pass the form URL as the first argument
  • Create a payload dict with form field names and values
  • The params argument sends the payload as query parameters
  • This automatically encodes and submits the form data. The response contains the results page based on the submitted search terms!

    Tips

  • Use developer tools to inspect form fields
  • Pay attention to field names and form method (GET or POST)
  • URL encode values as needed
  • Add headers to mimic a real browser
  • Learning to programmatically submit forms opens up many possibilities for web scraping and automation with Python requests!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!