Stories from the Web Crawling trenches in web servers

Efficient File Uploads in Python with aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

aiohttp provides a straightforward API for handling file uploads from clients. Validate and process uploads as byte streams. Check file headers for size/type before storage. Support multiple parallel uploads. Store uploaded files appropriately based on application needs.

Sending POST Requests with Python's urllib

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in Python provides functionality for sending HTTP POST requests to web servers and handling responses.

Easy Guide to Installing urllib in Python

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in Python allows you to open and read URLs. It is included in the Python standard library and works with Python 2.7.9+ and Python 3.4+. Import urllib.request to use it. Use urlopen() to make GET requests.

What is the difference between Httplib and Urllib?

Author: Mohan Ganesan

Date: Feb 20, 2024

Python code can make HTTP requests using urllib and httplib libraries. urllib is simpler and part of the standard library, while httplib provides more control and is suitable for advanced cases.

Properly Encode URLs in Python Requests with urllib

Author: Mohan Ganesan

Date: Feb 20, 2024

Properly encode URLs in Python using urllib to handle special characters, ensuring reliable transmission of HTTP requests.

Build High Performance Asyncio Web Servers in Python

Author: Mohan Ganesan

Date: Mar 25, 2024

Python's asyncio module allows you to write non-blocking, event-driven network code. This makes it possible to build very high performance web servers that can handle thousands of concurrent connections with very low resource usage.

Async IO in Python with aiohttp

Author: Mohan Ganesan

Date: Mar 3, 2024

aiohttp brings the performance benefits of async I/O to Python web development while retaining a simple, Pythonic API.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!