Skip to content

Comment on Applebot, the web crawler for Appleparent

Comments

"If robots instructions don't mention Applebot"

I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.

So a User-agent: * would be ignored if a User-agent: GoogleBot is found.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.