Skip to content

Comment on Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answersparent

Comments

For standard responses you're not wrong, but increasingly there are a lot of LLMS that are essentially doing RAG against search results. For example this is I believe how Kagi works, and Google AI overviews. I have increasingly seen Claude and chat GPT also doing the same thing where instead of answering a question from the knowledge Bank they will do a web search and cite the responses that were used.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.