Stories from the Web Crawling trenches in optimization

Speeding up Python Requests using gzip and other techniques

Author: Mohan Ganesan

Date: Dec 6, 2023

The Complete Guide to JavaScript Scraping with Python: Tips, Tricks, and Gotchas

Author: Mohan Ganesan

Date: Nov 17, 2023

Scraping JavaScript-heavy sites in Python can be tricky. With the right tools like Selenium and Requests-HTML, you can conquer complex JS pages and handle async JS rendering.

What is the fastest language for multithreading?

Author: Mohan Ganesan

Date: Mar 17, 2024

Multithreading improves performance. C++, Java, and Go are fastest. Optimize with thread pools, shared state, and reducing blocking.

Benchmarking aiohttp Web Performance

Author: Mohan Ganesan

Date: Feb 22, 2024

The Python aiohttp library provides powerful async HTTP client/server functionality. Benchmarking quantifies metrics like requests per second, latency distributions, and resource usage to guide optimization and capacity planning.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!