They could just timestamp different versions of robots.txt (which they probably do already), and respect it depending on date (which is more of a hassle, because you have to build it in your UI logic).
That would not solve the problem they're trying to solve.
Let's say I post something that I shouldn't have posted -- insider stock information, nude photos, whatever. Perhaps something illegal for me to post. I need to make it go away.
I need to be able to create a robots.txt today which affects stuff I posted yesterday.
This is why archive.org respects the current robots.txt for access to past content.
Comments
They could just timestamp different versions of robots.txt (which they probably do already), and respect it depending on date (which is more of a hassle, because you have to build it in your UI logic).
That would not solve the problem they're trying to solve.
Let's say I post something that I shouldn't have posted -- insider stock information, nude photos, whatever. Perhaps something illegal for me to post. I need to make it go away.
I need to be able to create a robots.txt today which affects stuff I posted yesterday.
This is why archive.org respects the current robots.txt for access to past content.