Stories from the Web Crawling trenches in GET requests

Passing Parameters in URLs with Python's urllib

Author: Mohan Ganesan

Date: Feb 8, 2024

Construct URL requests in Python using urllib module to pass parameters and handle encoding. GET requests use parameters in the URL, while POST requests use the request body.

Sending Data in GET Requests with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

The Python Requests library provides a simple way to send GET requests with data using the requests.get() method. It encodes the data into a query string that is appended to the URL, making it perfect for sending non-sensitive data like filters or pagination options.

Using Proxies with Axios in 2024

Author: Mohan Ganesan

Date: Jan 9, 2024

Learn how to integrate proxies with Axios for efficient web scraping and bot development. Avoid IP bans and scale your projects with ease.

Is Urllib built in Python?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python's urllib module provides the building blocks for fetching data and interacting with APIs over HTTP.

Making HTTP Requests in Python Without a Proxy

Author: Mohan Ganesan

Date: Feb 3, 2024

Make HTTP requests in Python without a proxy using the requests library. Customize requests with headers, parameters, and handle timeouts.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!