Stories from the Web Crawling trenches in parameters

Handling URL Encoding in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the Requests module, special characters in URLs can cause errors. The solution is to manually URL encode the parameters using quote_plus or the params argument.

Sending Parameters in URLs with the Python Requests Library

Author: Mohan Ganesan

Date: Feb 3, 2024

Making API requests with Python Requests library, passing parameters as a dictionary, handling URL parameters and headers for complex requests.

Passing Parameters in aiohttp Requests

Author: Mohan Ganesan

Date: Feb 22, 2024

Pass parameters in Python aiohttp requests using query string, form parameters, or JSON data to modify the response.

Passing Parameters in URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

Construct URL requests in Python using urllib module to pass parameters and handle encoding. GET requests use parameters in the URL, while POST requests use the request body.

Troubleshooting Bad Requests in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python requests module is invaluable for making HTTP requests in your code. Troubleshoot and fix 400 status errors by checking headers and parameters.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!