To me this feels almost like the news complaining that they want a "link tax." Weren't their headlines and summaries used? It seems inconsistent to somehow say that AI and scraping is not okay; but that news companies should also not be entitled to their link tax. It's okay to index, but not that kind of index.
It seems pretty cut-and-dry to me: the website owner should opt-out if they don't like the deal (being indexed in this case).
In the "link tax" case, there were plenty of trivial ways to opt out of headline usage - robots.txt, http headers, http tags. The problem was newspapers did not want to opt out (as they were benefiting from Google themselves), so they wanted a 3rd option. Which was pretty stupid of course - if you don't like the deal, don't take it; suing the offering party for a better deal is not a good long-term strategy.
In the AI case, there is no opt-out. All those websites already indicated they want to opt-out via robots.txt, but the AI companies ignore robots.txt, change user-agent, fake IPs, and so on - do the things that are normally done by shady malwar-ish services rather than multi-billion-dollar companies.
It really bothers me when people don't see the difference between those two cases.
Comments
I agree even if I have mixed feelings on it.
To me this feels almost like the news complaining that they want a "link tax." Weren't their headlines and summaries used? It seems inconsistent to somehow say that AI and scraping is not okay; but that news companies should also not be entitled to their link tax. It's okay to index, but not that kind of index.
It seems pretty cut-and-dry to me: the website owner should opt-out if they don't like the deal (being indexed in this case).
In the "link tax" case, there were plenty of trivial ways to opt out of headline usage - robots.txt, http headers, http tags. The problem was newspapers did not want to opt out (as they were benefiting from Google themselves), so they wanted a 3rd option. Which was pretty stupid of course - if you don't like the deal, don't take it; suing the offering party for a better deal is not a good long-term strategy.
In the AI case, there is no opt-out. All those websites already indicated they want to opt-out via robots.txt, but the AI companies ignore robots.txt, change user-agent, fake IPs, and so on - do the things that are normally done by shady malwar-ish services rather than multi-billion-dollar companies.
It really bothers me when people don't see the difference between those two cases.