Stories from the Web Crawling trenches in permissions

Can I crawl any website?

Author: Mohan Ganesan

Date: Feb 20, 2024

When creating a web crawler, it is important to respect websites' permissions and crawl ethically. The Robots Exclusion Protocol and proper identification of the crawler are key factors. Legal risks can be avoided by obtaining explicit permission from website owners.

Troubleshooting Error Code 1 When Installing aiohttp Python Package

Author: Mohan Ganesan

Date: Mar 3, 2024

Error code 1 when installing aiohttp or other Python packages with native C code can be caused by missing dependencies, incorrect gcc version, permissions issue, or corrupted build.

Tired of getting blocked while scraping the web?

ProxiesAPI handles headless browsers and rotates proxies for you.
Get access to 1,000 free API credits, no credit card required!