Comment on Facebook's robots.txtComments−yalogin12ySo what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?−dblacc12yI could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).−elbear12yYou are correct.
Comments
So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?
I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).
You are correct.