Comment on Kogan.com mysteriously disappears from Microsoft Search EnginesComments−UnoriginalGuy14yTheir robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html"I have no idea what consequences this might have since I've never seen another site which does it. Some quick Googling turned up nothing.Maybe Bing choked on the wrong content type and decided to blackball their entire domain in order to avoid indexing the unindexable?−X-Istence14yWhy would you want to serve up a robot.txt as "text/html" when in reality it is "text/plain"?
Comments
Their robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html"
I have no idea what consequences this might have since I've never seen another site which does it. Some quick Googling turned up nothing.
Maybe Bing choked on the wrong content type and decided to blackball their entire domain in order to avoid indexing the unindexable?
Why would you want to serve up a robot.txt as "text/html" when in reality it is "text/plain"?