Stories from the Web Crawling trenches in POST request

Python Requests Cheatsheet

Author: Mohan Ganesan

Date: Jan 9, 2024

Overview of Requests, a popular HTTP library for Python. Features include making GET and POST requests, handling response content and headers.

Uploading Zip Files via HTTP POST with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Sending zip files over HTTP using Python's Requests library with multipart form data for efficient file upload and server processing.

Simplifying HTTP Requests in Python: Urllib vs. Requests

Author: Mohan Ganesan

Date: Feb 8, 2024

When working with HTTP requests in Python, you have two options: urllib or requests. urllib is low-level but built-in, while requests is simple and intuitive. Use requests for typical tasks and urllib for fine-grained control.

Automate Website Logins with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Logging into websites made easy with Python's requests module. Replicate login process, handle response codes, automate workflows.

Making HTTP Requests in Python: requests vs. pycurl

Author: Mohan Ganesan

Date: Feb 3, 2024

Python provides options for making HTTP requests. Use requests library for basic needs and pycurl for more control.

Why My Python requests.post() is Sending a GET Instead of POST

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with Python's popular requests library, calling requests.post() may send a GET request instead of POST due to forgetting to pass data/json or server redirection.

Making HTTP Requests in Python with HTTPX

Author: Mohan Ganesan

Date: Feb 5, 2024

Python HTTP client HTTPX simplifies making HTTP requests, supports HTTP/1.1 and HTTP/2, and offers features like timeouts and retries.

Sending POST Requests with the Python Requests Library by Specifying GET

Author: Mohan Ganesan

Date: Feb 3, 2024

Override the method parameter in Python Requests library to make a POST request even if specified as GET.

urllib get

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides a simple interface for fetching data over HTTP. With just a few lines of code, you can easily make GET and POST requests to access web pages and APIs.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!