Skip to content

Comment on Robots.txt for the NYT has a specific exclusion for an 1996 news article

Comments

the US Department of State (DoS) during 2012/2013 in its robots.txt[1] excluded around 9577 documents which leaked into archive.org (already pre Snowden). The robots.txt file now is OK but not sure if content is still on archive.

[1] https://pastebin.com/raw/RE2tpyR3

  8<-----------8<-----------8<-----------8<-----------8<-----------

  #!/bin/bash
  snapshots="20120713050942 20121013154343 20121010165822 20120921054221 20130413152313 20130113162428"
  # orig source http://state.gov/robots.txt but also on pastebin in case they delete it:
  wget --output-document=robots.txt http://pastebin.com/raw.php?i=RE2tpyR3
  for x in `echo $snapshots`
  do
    for i in `cat ./robots.txt|cut -d ' ' -f2 | tr -d '\15\32'`
    do
      if [ -e `basename $i` ]; then
        echo "$i already fetched"
      else
        wget https://web.archive.org/web/$x/http://www.state.gov/documents/$i;
      fi
    done
  done
AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.