Stories from the Web Crawling trenches in GET request

Python Requests Cheatsheet

Author: Mohan Ganesan

Date: Jan 9, 2024

Overview of Requests, a popular HTTP library for Python. Features include making GET and POST requests, handling response content and headers.

Getting Started with HTTPX in Python: Practical Examples and Usage Tips

Author: Mohan Ganesan

Date: Feb 5, 2024

HTTPX is a powerful Python HTTP client that makes API calls, handles authentication, timeouts, and more. Easily make GET and POST requests, handle JSON, forms, files, and headers. Supports async requests and session reuse for optimal performance.

Sending GET Requests with Python Requests using Postman

Author: Mohan Ganesan

Date: Feb 3, 2024

Postman is a popular API testing tool that allows you to easily make HTTP requests. This article explains how to make a simple GET request using Python's requests library and Postman, and how to process the JSON response.

Easy Guide to Installing urllib in Python

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in Python allows you to open and read URLs. It is included in the Python standard library and works with Python 2.7.9+ and Python 3.4+. Import urllib.request to use it. Use urlopen() to make GET requests.

Simplifying HTTP Requests in Python: Urllib vs. Requests

Author: Mohan Ganesan

Date: Feb 8, 2024

When working with HTTP requests in Python, you have two options: urllib or requests. urllib is low-level but built-in, while requests is simple and intuitive. Use requests for typical tasks and urllib for fine-grained control.

Fetching Data from APIs with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Accessing data from web APIs using Python's Requests library. Learn how to make GET requests, process responses, and handle errors.

Troubleshooting Python Requests Get When Webpage Isn't Loading

Author: Mohan Ganesan

Date: Feb 3, 2024

When using Python's Requests library to load a webpage, troubleshoot by checking the URL, status code, response headers.

Making HTTP Requests in Ruby with the httpx Gem

Author: Mohan Ganesan

Date: Feb 5, 2024

The httpx gem provides a simple and flexible way to make HTTP requests in Ruby, with features like persistent connections and timeouts. It's great for APIs, web scraping, and tasks involving HTTP requests.

Making HTTP Requests in Python: requests vs. pycurl

Author: Mohan Ganesan

Date: Feb 3, 2024

Python provides options for making HTTP requests. Use requests library for basic needs and pycurl for more control.

Making HTTP Requests in Python with HTTPX

Author: Mohan Ganesan

Date: Feb 5, 2024

Python HTTP client HTTPX simplifies making HTTP requests, supports HTTP/1.1 and HTTP/2, and offers features like timeouts and retries.

Sending POST Requests with the Python Requests Library by Specifying GET

Author: Mohan Ganesan

Date: Feb 3, 2024

Override the method parameter in Python Requests library to make a POST request even if specified as GET.

Making HTTP Requests with aiohttp in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

The aiohttp library is a popular asynchronous HTTP client/server framework for Python. It allows you to make HTTP requests without blocking your application, perfect for building highly concurrent or asynchronous services.

Why My Python requests.post() is Sending a GET Instead of POST

Author: Mohan Ganesan

Date: Feb 3, 2024

When working with Python's popular requests library, calling requests.post() may send a GET request instead of POST due to forgetting to pass data/json or server redirection.

urllib get

Author: Mohan Ganesan

Date: Feb 8, 2024

The urllib module in Python provides a simple interface for fetching data over HTTP. With just a few lines of code, you can easily make GET and POST requests to access web pages and APIs.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!