Yeah I've been really surprised by how inept they are. Lots of others have fallen for the trap but extract themselves fairly quickly and probably blacklist my site. Archive.org got stuck for a while, which was unfortunate, so I had to exclude them manually. There's something on an Oracle network that keeps coming back but it's very slow compared to the ridiculous rate of requests from Meta's scraper.
It's up to 1.5 million requests now, sigh. My guess is that Meta has too much money and is in the "move fast and break things" stage where they're just throwing money at a problem. Not setting a max crawl depth is hilarious though.
Comments
Yeah I've been really surprised by how inept they are. Lots of others have fallen for the trap but extract themselves fairly quickly and probably blacklist my site. Archive.org got stuck for a while, which was unfortunate, so I had to exclude them manually. There's something on an Oracle network that keeps coming back but it's very slow compared to the ridiculous rate of requests from Meta's scraper.
It's up to 1.5 million requests now, sigh. My guess is that Meta has too much money and is in the "move fast and break things" stage where they're just throwing money at a problem. Not setting a max crawl depth is hilarious though.