Stories from the Web Crawling trenches in caching

Speeding up Python Requests using gzip and other techniques

Author: Mohan Ganesan

Date: Dec 6, 2023

Caching in Python

Author: Mohan Ganesan

Date: Dec 6, 2023

Learn how to cache API responses in Python to improve performance. Caching reduces API requests, improves speed, and lowers costs.

Using aiohttp for Easy and Powerful Reverse Proxying in Python

Author: Mohan Ganesan

Date: Mar 3, 2024

Reverse proxying with aiohttp in Python allows for load balancing, caching, security, and more. ProxyResolver and ProxyConnector provide customization options.

The Ultimate Select.rs Cheat Sheet for Rust

Author: Mohan Ganesan

Date: Oct 31, 2023

select.rs is a robust HTML/XML scraping library for Rust. This cheat sheet covers its features, including installation, loading documents, selecting nodes, traversing nodes, extracting/modifying nodes, creating/inserting/removing nodes, output formats, caching and persistence, headless browsers, validation, encoding, advanced selectors, caching and performance, common recipes, troubleshooting, and ecosystem libraries.

Controlling HTTP Requests with urllib Headers

Author: Mohan Ganesan

Date: Feb 6, 2024

The Python urllib module provides a powerful way to make HTTP requests in your code. Headers allow you to specify important metadata about the request, like the user agent, authentication credentials, caching settings, and more.

Configuring Headers with aiohttp Clients for Effective API Calls

Author: Mohan Ganesan

Date: Feb 22, 2024

Properly configuring headers in aiohttp is crucial for smooth API requests. Headers serve purposes like authentication, context, security, and caching.

Troubleshooting Stale Data in Python Requests

Author: Mohan Ganesan

Date: Feb 3, 2024

Check for client-side caching in requests and disable. Ensure server is not caching responses. Use sessions for APIs that require statefulness.

Is Beautiful soup slow?

Author: Mohan Ganesan

Date: Feb 5, 2024

Beautiful Soup is a popular Python library for web scraping. It can be optimized for faster scraping by using appropriate parsers, parsing only necessary data, caching, and performance profiling.

Speed Up Python Requests with Caching

Author: Mohan Ganesan

Date: Feb 3, 2024

HTTP requests in Python using requests library can be faster due to caching. Caching avoids unnecessary work and streamlines data retrieval workflows.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!