chrislomax on Kogan.com mysteriously disappears from Microsoft Search Engines

Comment on Kogan.com mysteriously disappears from Microsoft Search Engines

Anyone else thinking they added a robots.txt entry for bing to disallow, waited to get de-listed then removed the offending robots.txt entry, posted this news article knowing bing would pick them back up in a couple of days?

michaelt 14y

The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified: Wed, 06 Jun 2012 06:14:35 GMT

It's possible they did this and they were cunning enough to turn back the last-modified clock.

reinhardt 14y

How did you get that? Curl says May 4 instead:

  $ curl -I http://www.kogan.com/robots.txt
  HTTP/1.1 200 OK
  Server: nginx/1.0.14
  Date: Mon, 09 Jul 2012 18:48:34 GMT
  Content-Type: application/octet-stream
  Content-Length: 79
  Last-Modified: Fri, 04 May 2012 18:58:02 GMT
  Connection: keep-alive
  Accept-Ranges: bytes

michaelt 14y

I get Last-Modified: Fri, 04 May 2012 18:58:02 GMT from my home machine (and you obviously do too) where I connect to 74.118.68.224 - but I got Wed, 06 Jun 2012 06:14:37 GMT from work (and via http://web-sniffer.net/ and http://www.rexswain.com/cgi-bin/httpview.cgi) which was looking at 108.166.39.66. Perhaps we get DNS load balanced or something similar?

jbigelow76 14y

If that's what they did (no telling really) that would be pretty clever PR wise.

rmc 14y

Clever… and dishonest.

taligent 14y

Very much Kogan's style.

Comments