Stories from the Web Crawling trenches in user experience

Does YouTube allow scraping?

Author: Mohan Ganesan

Date: Feb 20, 2024

YouTube restricts data scraping to protect its platform and users. Exceptions include limited personal use and research purposes.

Securely Share Sessions Between Services with Aiohttp Session Proxy

Author: Mohan Ganesan

Date: Feb 22, 2024

Aiohttp session proxy allows secure sharing of session data between microservices, improving user experience and ensuring encryption. Best practices include setting environment variables, using HTTPS, and handling timeouts.

httpnotfound aiohttp

Author: Mohan Ganesan

Date: Feb 22, 2024

Handle 404 errors in web applications using Python aiohttp framework, with custom error handler and templating for a better user experience.

Is asyncio part of Python?

Author: Mohan Ganesan

Date: Mar 17, 2024

Python's asyncio module enables non-blocking concurrency, improving performance, scalability, and user experience.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!