How Web Servers Stop Bots?
Many web servers mainly want to allow humans to query them and disallow bots.
Here are the main ways how the web servers stop bots from doing their thing:
They look at the User-Agent-String header sent by the bot. If it is not a familiar one sent by a known browser on a known OS, they might reject the call.
The bot is making too many calls too frequently. Web servers know that humans dont do that and can block you out.
The bot is making calls in a sequential order which humans rarely do.
The bot is making calls at the same time every day. The call logs show it and the webmaster can block you.
The bot is making calls at predictable intervals.
The bot is not passing the cookies back. Even though the cookies are not relevant to the use of the data, the web servers expect it to come back as all browsers do pass them back.
The User-Agent-String is always the same and is not rotated.
