The short answer to this question is. It depends.
"Depends on that?" you ask.
Well, it depends on your intent. How did you get here? Look around you. How did you get on this beautiful blog post? You got here from Google, which scraped MY DATA. But it helps me.
The intent of Google is just fine. It also makes money off of the services rendered, which I am fine with as well.
What about ethics?
What about it? Again it's intent. Suppose you download a few pictures of cats and entertain your kids with it, it's entirely ethical. But if the original owner wanted to make money off of it by selling it as quality photography and you started re-selling it on your website, that's just bad manners, and you should probably get sued.
Its also how you do it.
If you use a super-powerful bot to scrape a website that is not designed to handle all that many concurrent requests and bring down their web server, you are being mean and create a bad vibe for yourself and all web scrapers in general. Respect the users and the administrators of the website and try and make it as easy on them as possible.
Some people are panicky.
Some people are opposed to even Google scraping them. I would love to be one of those people and that cranky when I get to 90 years of age. So what should Google do to let them opt-out? Give them space to opt-out. Yeah. And so should you.
But even this is a grey area. You can argue that any data aired in public should be scrapable. Quite recently, sometime in the last quarter of last year, federal courts weighed in on this issue and simply declared that Scraping Public Data Doesn't Violate Hacking Laws. more here
Web Scraping legalities from the QVC case
Also, here is how to.
The grey area of Web Scraping Legalities Explored
In any case, if you are interested in web scraping and need to be sure not to get blocked by any web server, you will need a Rotating Proxy Service.
Our rotating proxy service Proxies API provides a simple API that can solve all IP Blocking problems instantly.
- With millions of high speed rotating proxies located all over the world
- With our automatic IP rotation
- With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
- With our automatic CAPTCHA solving technology
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.
A simple API can access the whole thing like below in any programming language.
curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"We have a running offer of 1000 API calls completely free. Register and get your free API Key here.
