Stories from the Web Crawling trenches in Python applications

Understanding HTTP Status Codes with Python Requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Making HTTP requests is a fundamental task in many Python applications. HTTP status codes provide meaningful insight into API responses. Handle different status code classes properly in your application.

Uploading Files in Python Requests: A Guide

Author: Mohan Ganesan

Date: Feb 3, 2024

Sending file uploads via HTTP requests is a common task in many Python applications. This guide covers how to upload files using the requests library and multipart/form-data.

Effective Strategies for Rate Limiting Asynchronous Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Making asynchronous HTTP requests in Python applications and effectively rate limiting them using queues, retrying failed requests with backoff, and monitoring usage to stay under limits.

Debugging HTTP Requests with httpx Debug

Author: Mohan Ganesan

Date: Feb 5, 2024

Making HTTP requests is core functionality for many Python applications. httpx debug is a debugging proxy server that captures HTTP traffic, logs request/response data, and allows for mocking and modifying traffic for testing scenarios.

Handling User Input in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Handle user input in Python applications with the requests library. Get textual and numeric input, upload files, and handle sensitive inputs like passwords. Validate dangerous inputs to avoid security issues.

Simplifying HTTP Requests in Python: urllib2 vs urllib vs requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python: from urllib2 to requests. urllib2 - Python's Default HTTP Client. urllib - A Minor Improvement. requests - A Simple Yet Powerful Library.

Smarter Retries with Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Adding smart retries to Python requests improves reliability by using exponential backoff and handling exceptions separately.

Troubleshooting Slow and Failing Requests in Python

Author: Mohan Ganesan

Date: Feb 3, 2024

Making HTTP requests in Python applications can sometimes be problematic. This article provides tips for troubleshooting slow or failing requests, including checking for network/server issues, setting sensible timeouts, inspecting the request object, and profiling long requests.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!