The 6 commandments of web scraping
The 6 commandments of web scraping are:
- Though shalt always use a framework.
- Though shalt use CSS selectors or XPath for web scraping.
- Though shalt build monitors that check for expected results in your crawler.
- Though shalt imitate humans at every turn.
- Though shalt use a rotating proxy service.
- [Optional] Though shalt use these 4 external services to stabilize your web crawler.
