Skip to content

Comment on Robots.txt for the NYT has a specific exclusion for an 1996 news article

Comments

Just for future reference: adding a URL to robots.txt will not necessarily exclude that web page from Google, especially if it has already been indexed.

To reliably exclude a URL from indexing, you have to serve a “no index” instruction with that URL, either in a meta tag or an HTTP header. And for this instruction to be read, the robot has to visit that page! So disallowing the URL in robots.txt can actually be counterproductive to de-indexing it.

Google also offers a tool specifically for removing URLs from their index in Search Console.

That's right, but the url exclusion tool is only temporary, so the only way to do it correctly is with the noindex tag.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.