As a web crawler, the web is a scary place. It is like landing a spaceship on an asteroid. It is because the original web browsers like Internet Explorer were very lenient on poorly written HTML, so many of the pattern matching techniques keep failing. You learn to appreciate technologies like Beautiful Soup to help tame rouge HTML.
2. It helps you understand web servers.
Web servers use several ‘tells’ to detect that you are not human. It is a bit like a game of Cat and Mouse.
3. It might teach you a new language
If you want pure power, sooner or later, you will realize the power of Scrapy, especially when combined with Scrapyd. A long, long time ago, I had to learn Python to get all the advantages Scrapy gives me, which is easily offset by the disadvantages of having to learn something new.
4. It tests your ability to think out of the box
Each website is different, has AJAX-based content; pagination comes in all shapes and sizes. You have to ‘crack’ each website separately many times as there is no one size fits all.
5. It challenges you to work at scale
What works when you are in the development mode very often crash lands when you run it at scale. To use concurrency, scale spiders, handle large amounts of data, maximize the throughput of a server or several servers, use rotating proxies to overcome IP blocks like Proxies API. Coding is only the first step.
6. It allows you to tame the chaos
It is what most of your code does. You are trying to get data out of something that wasn’t built for your bots. Most of the time, it is not easy to domesticate them.
7. It teaches you to work with frameworks
Scrapy on Scrpayd along with file pipelines if you have to download images. CSS/XPath Selectors to make scraping more predictable. Rate limiting, session handling, and concurrency — You will learn all this by trial and error.
8. It teaches you the importance of creating checks and balances
The author is the founder of Proxies API, a proxy rotation api service.