Stories from the Web Crawling trenches in use cases

Is VPN good for scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

Using a VPN for web scraping can provide privacy and access benefits, but it may also slow down page load times and have usage limits.

Choosing Between Curio and aiohttp for Async IO in Python

Author: Mohan Ganesan

Date: Feb 22, 2024

Python developers can choose between Curio and aiohttp for async IO. Curio is great for CPU-bound tasks, while aiohttp is ideal for IO-bound HTTP applications. Both libraries are well-optimized for performance.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!