Stories from the Web Crawling trenches in efficiency

Making Partial Updates with PATCH Requests in Python

Author: Mohan Ganesan

Date: Nov 17, 2023

PATCH requests allow partial updates to resources via APIs. Python's requests module makes it easy to send PATCH requests and modify specific attributes using JSON patch docs.

Whats the equivalent of pythons request package for rust?

Author: Mohan Ganesan

Date: Feb 3, 2024

Rust is a systems programming language focused on performance, reliability, and efficiency. reqwest is a popular HTTP client library for Rust, providing a similar developer experience to Python's requests package.

urllib Connection Pool in Python

Author: Mohan Ganesan

Date: Feb 8, 2024

Using a connection pool in Python's urllib module is a best practice for making multiple requests, boosting efficiency and speed.

Asyncio event loop

Author: Mohan Ganesan

Date: Mar 25, 2024

The asyncio module is a powerful tool for writing concurrent and asynchronous code. The event loop manages tasks and callbacks, allowing for efficient handling of thousands of concurrent requests.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!