Why wouldn't it consider the archived state of robots.txt?