Stories from the Web Crawling trenches in web requests

Fetching Web Resources with urllib in MicroPython

Author: Mohan Ganesan

Date: Feb 6, 2024

The urllib module in MicroPython provides a simple interface for fetching resources from the web. It can handle HTTP requests and responses, making it easy to fetch JSON data, download images, and more.

Handling User Input in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Handle user input in Python applications with the requests library. Get textual and numeric input, upload files, and handle sensitive inputs like passwords. Validate dangerous inputs to avoid security issues.

Scraping Reddit Posts with Rust

Author: Mohan Ganesan

Date: Jan 9, 2024

Code walkthrough for scraping Reddit using Rust to extract post information.

Asyncio gathering task results

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio.gather() function is useful for launching multiple coroutines concurrently and waiting for their results. It is commonly used for coordinating web requests, IO work, and parallel flows.

Is asyncio a standard library python ?

Author: Mohan Ganesan

Date: Mar 17, 2024

Async IO is a useful concurrent programming framework in Python's standard library for executing multiple tasks concurrently within a single thread.

urllib retrieve

Author: Mohan Ganesan

Date: Feb 8, 2024

urllib in Python makes it easy to fetch resources from the web. Handle errors and include data in requests with URL encoding.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!