Archive has actually lost some of the archives I stored on it
Who currently controls the domain? One of the ways stuff can be lost is if a new domain owner fiddles with robots.txt. (The Archive has recently changed their policy about that)
We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine.
Is there a delay between the time the robots.txt changes and the time when the content becomes inaccessible via Wayback Machine? How often does the archive.org_bot crawl robots.txt?
Can a script check robots.txt periodically for changes and if changes are detected, then download the content from Wayback Machine before it becomes inaccessible?
Additionally, can a script check the domain registration for an anticipated expiration date, or perhaps monitor domainname "drop lists"?
Comments
Archive has actually lost some of the archives I stored on it
Who currently controls the domain? One of the ways stuff can be lost is if a new domain owner fiddles with robots.txt. (The Archive has recently changed their policy about that)
We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Is there a delay between the time the robots.txt changes and the time when the content becomes inaccessible via Wayback Machine? How often does the archive.org_bot crawl robots.txt?
Can a script check robots.txt periodically for changes and if changes are detected, then download the content from Wayback Machine before it becomes inaccessible?
Additionally, can a script check the domain registration for an anticipated expiration date, or perhaps monitor domainname "drop lists"?