Stories from the Web Crawling trenches in response handling

Setting the Content-Type Header for Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Properly setting the Content-Type helps the receiving server interpret and handle the data correctly. When sending JSON data or other formats, you'll want to explicitly set the header instead. Uploading multipart form data requires setting the content type accordingly. Handling responses and content types appropriately is important for robust integrations.

Understanding HTTP Status Codes with Python Requests

Author: Mohan Ganesan

Date: Oct 22, 2023

Making HTTP requests is a fundamental task in many Python applications. HTTP status codes provide meaningful insight into API responses. Handle different status code classes properly in your application.

Troubleshooting aiohttp ServerDisconnectedError

Author: Mohan Ganesan

Date: Feb 22, 2024

If you're using Python's aiohttp library for asynchronous HTTP requests and getting ServerDisconnectedErrors, here are some troubleshooting tips to handle the response inside the context manager and check for connectivity issues.

Fetching Web Resources with urllib in MicroPython

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in MicroPython provides a simple interface for fetching resources from the web. It can handle HTTP requests and responses, making it easy to fetch JSON data, download images, and more.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!