Comment on PSA – ByteDance and Bytespider Bots – Recommend BlockingComments−bediger40003yThe article is correct: Bytespider does not observe robots.txt directives.I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly behaved.
Comments
The article is correct: Bytespider does not observe robots.txt directives.
I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly behaved.