After the greader*-grab programs upload data to the target server, it is removed from your machine. All of the data eventually ends up in WARCs at https://archive.org/details/archiveteam_greader
As for an API, someone will hopefully write one to directly seek into a megawarc in that archive.org collection, or import everything into their feed reading service.
Comments
After the greader*-grab programs upload data to the target server, it is removed from your machine. All of the data eventually ends up in WARCs at https://archive.org/details/archiveteam_greader
As for an API, someone will hopefully write one to directly seek into a megawarc in that archive.org collection, or import everything into their feed reading service.
Any way I could patch the program to stop it from deleting the data after it is uploaded?
Among other things I would like to set up an ElasticSearch cluster for my own feeds.
Is the WARC format defined somewhere? I haven't looked at any other ArchiveTeam projects so I'm not informed if this format is used elsewhere.
Yeah, you could patch seesaw-kit to not delete local data. Note that greader-grab just gets a random work item from the tracker.
There's an ISO spec for WARC and tools linked at http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...
I haven't tried it, but the --keep-data option might work?
https://github.com/ArchiveTeam/seesaw-kit/blob/master/run-pi...
Yeah, that looks like the right thing.