The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record.
I thought kogan.com was around in 2008 as the blog goes back to 2008.
The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp).
I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.
Comments
You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009.
In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).
The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record.
I thought kogan.com was around in 2008 as the blog goes back to 2008.
The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp).
I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.