Oh, I wasn't saying you can completely stop them from reading a page or individual pages. But there is activity, than can be detected as irregular. Here is a true example I know about someone who wanted to scrape their competitor's client listings. The competitor had a map with points of their customers with random user IDs and no where was the entire dataset visible. The person just built a scraper/bot, to hit every single possible ID of over 10,000 numbers. They hit a ton of empty pages, and that company should have recognized an IP incrementally crawling their data, especially empty pages...This activity should have been recognized and resulted in an IP ban.
Comments
Oh, I wasn't saying you can completely stop them from reading a page or individual pages. But there is activity, than can be detected as irregular. Here is a true example I know about someone who wanted to scrape their competitor's client listings. The competitor had a map with points of their customers with random user IDs and no where was the entire dataset visible. The person just built a scraper/bot, to hit every single possible ID of over 10,000 numbers. They hit a ton of empty pages, and that company should have recognized an IP incrementally crawling their data, especially empty pages...This activity should have been recognized and resulted in an IP ban.