Comment on Facebook's robots.txtparentComments−toomuchtodo12yYou just ignore the robots.txt file, crawl slowly, and from distributed virtual machines.Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.
Comments
You just ignore the robots.txt file, crawl slowly, and from distributed virtual machines.
Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.