Skip to content

Comment on Ask HN: A good list of bots/crawlers ip addresses?

Comments

Another option would be to use robots.txt to stop bots from accessing a particular URL (for example, an 1x1 image or somesuch). Hide that somewhere in every page, and only count visits where the image was shown.

This does require that the url expansion works as a display + redirect, so an intermediate page is shown. If it doesn't work like that...

Well, you can simply exclude the bots and crawlers with robots.txt. The downside of that is that then they won't index your shortened links either, which may or may not be a problem.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.