Googlebot also does some other crazy stuff. Like looking at url patterns and then trying out variations.. they're almost trying to sniff URLs!
For example if I have a page:
www.domain.com/xyz/123
Googlebot (without any links to other pages, will actually try URLs like)
www.domain.com/xyz/1234
www.domain.com/xyz/122
www.domain.com/xyz/121
and so on...
It's crazy how much 'looking around' they do these days!
Comments
Googlebot also does some other crazy stuff. Like looking at url patterns and then trying out variations.. they're almost trying to sniff URLs!
For example if I have a page: www.domain.com/xyz/123
Googlebot (without any links to other pages, will actually try URLs like) www.domain.com/xyz/1234 www.domain.com/xyz/122 www.domain.com/xyz/121 and so on...
It's crazy how much 'looking around' they do these days!
I believe that one's mostly a search for duplicate content - looking for URL parameters that don't make a difference.