Stories from the Web Crawling trenches in requests module

How to Authenticate with Bearer Tokens in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Bearer tokens are used for authentication in APIs. This article explains how to make authenticated requests with bearer tokens in Python using the Requests module.

Handling URL Encoding in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the Requests module, special characters in URLs can cause errors. The solution is to manually URL encode the parameters using quote_plus or the params argument.

Troubleshooting the WinError 10061 with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Encountering WinError 10061 when using Python's requests module? Check for firewall issues, verify TLS versions, and ensure proper name resolution.

Fetching the Server IP Address with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Fetch and validate server IPs with Python Requests for monitoring, security, analytics, and troubleshooting purposes.

Making Partial Updates with PATCH Requests in Python

Author: Mohan Ganesan

Date: Nov 17, 2023

PATCH requests allow partial updates to resources via APIs. Python's requests module makes it easy to send PATCH requests and modify specific attributes using JSON patch docs.

Sending Text Data in a POST Request with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Guide on how to send plain text data in POST requests using Python requests module and setting Content-Type header.

How to fix MissingSchema error in Python requests

Author: Mohan Ganesan

Date: Oct 22, 2023

The MissingSchema error occurs when making a request to a URL without specifying the protocol. This article explains the causes of the error and provides various ways to fix and handle it properly.

Sending JSON vs Form Data in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

When making HTTP requests in Python using the requests module, you can send request bodies in different formats like JSON or form-urlencoded data.

The Redirect Ninja's Guide to Mastering Python Requests

Author: Mohan Ganesan

Date: Oct 31, 2023

Learn how to handle redirects in web scraping using Python's Requests module. Master techniques like sessions, custom redirect handlers, and inspecting redirects.

Downloading Binary Files with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Python's requests module makes it easy to download binary files from the internet. Learn how to stream the download and display a progress bar for efficient downloading.

Why You May Not Get All Cookie Data with the Python Requests Module

Author: Mohan Ganesan

Date: Feb 3, 2024

Use Sessions or custom jars to ensure you have full cookie details when using Requests.

Python requests vs urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

Python provides two modules for making HTTP requests: requests and urllib. Requests simplifies HTTP calls while urllib provides more flexibility.

Speed Up Your Website: Measuring Page Load Times in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Measure page load times in Python using the requests module to provide a good user experience. Fetch a webpage and calculate the duration it takes to fully load.

Automate Website Logins with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Logging into websites made easy with Python's requests module. Replicate login process, handle response codes, automate workflows.

Automating Web Interactions in Python with Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Automate web interactions with Python Requests library. Easily submit forms, scrape data, and click buttons programmatically.

Persisting Cookies from Initial Request in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Save and re-use cookies in Python requests. Use cookies for session state and authentication. Save cookies to variable or use a session for automatic cookie persistence.

Troubleshooting "ImportError: No module named requests" in VS Code

Author: Mohan Ganesan

Date: Feb 3, 2024

Frustrated with ImportError in VS Code? Check Python interpreter, reinstall requests, use virtual environment. Restart VS Code for changes to take effect.

Simplifying REST API Calls with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Interacting with REST APIs made easy with Python's Requests module. Simple syntax, JSON decoding, parameterization, and more. Try it now!

What is Urllib Python?

Author: Mohan Ganesan

Date: Feb 20, 2024

Urllib is a Python library for making HTTP requests and working with URLs. It is useful for basic requests and simple GET requests. For more advanced functionality, consider using the requests module and other 3rd party packages.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!