Comment on Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answersparentComments−brookst1moSo an HTML page with a list of essentially every page on every site on the internet? What do you think, maybe a trillion URLs?−amelius1moYes. Whatever makes their lawyers nervous.
Comments
So an HTML page with a list of essentially every page on every site on the internet? What do you think, maybe a trillion URLs?
Yes. Whatever makes their lawyers nervous.