There have been collaborative computing projects like SETI@home [1] and Folding@Home [2] where unused computing power could be used for productive purposes. Could there be something similar for storage? Software that provides unused storage for Internet archiving? In the best case scenario, we could have redundant backups of the Internet Archive distributed around the world.
archive.org does use torrents and I have one such torrent laying around in my client, which occasionally connects to peers although the the trackers are currently offline. I suppose a new client would find me and other peers through the DHT. I'd share a magnet link for someone to try, but it's a copyright-ignoring ROM dump archive so it may not be the best idea to post it here.
It's interesting that torrents may not be the first thing that comes to mind. They have the "PR issue" of being the now seemingly mundane way in which we've been downloading DVD rips for the last 20 odd years. Newer technology like IPFS does a better job making the cool core of this technology actually sound cool.
I didn’t know that archive.org already has torrents. I guess what we would need, on top of that, is a system for assigning those torrents to new peers.
That already exists. Peers find each other through a distributed hash table which can be bootstrapped from a variety of sources.
I would say the problem is discoverability and actual deployment.
For suddenly popular files it can be a way to donate bandwidth most of all, because then there might suddenly be a lot of peers. For the vast majority of files however, there won't be any other peers and they have to be web seeded by Archive.org either way.
Then there's the discoverability problem. Ultimately you need something like a magnet link to connect to a swarm.
IPFS on the tin seems pretty awesome however when I attempted to dig into it for an hour I still had no idea how to actually do anything with it. Their usability needs to go a long way before I give it another go. it's definitely not a two step process where you download a client and click on a link to start load sharing an archive in my past experience.
Comments
There have been collaborative computing projects like SETI@home [1] and Folding@Home [2] where unused computing power could be used for productive purposes. Could there be something similar for storage? Software that provides unused storage for Internet archiving? In the best case scenario, we could have redundant backups of the Internet Archive distributed around the world.
[1]: https://setiathome.berkeley.edu/
[2]: https://foldingathome.org/
Perhaps torrents?
archive.org does use torrents and I have one such torrent laying around in my client, which occasionally connects to peers although the the trackers are currently offline. I suppose a new client would find me and other peers through the DHT. I'd share a magnet link for someone to try, but it's a copyright-ignoring ROM dump archive so it may not be the best idea to post it here.
It's interesting that torrents may not be the first thing that comes to mind. They have the "PR issue" of being the now seemingly mundane way in which we've been downloading DVD rips for the last 20 odd years. Newer technology like IPFS does a better job making the cool core of this technology actually sound cool.
I didn’t know that archive.org already has torrents. I guess what we would need, on top of that, is a system for assigning those torrents to new peers.
That already exists. Peers find each other through a distributed hash table which can be bootstrapped from a variety of sources.
I would say the problem is discoverability and actual deployment.
For suddenly popular files it can be a way to donate bandwidth most of all, because then there might suddenly be a lot of peers. For the vast majority of files however, there won't be any other peers and they have to be web seeded by Archive.org either way.
Then there's the discoverability problem. Ultimately you need something like a magnet link to connect to a swarm.
https://github.com/anacrolix/btlink
The vision behind IPFS is that (to an extent) https://ipfs.tech/
IPFS on the tin seems pretty awesome however when I attempted to dig into it for an hour I still had no idea how to actually do anything with it. Their usability needs to go a long way before I give it another go. it's definitely not a two step process where you download a client and click on a link to start load sharing an archive in my past experience.
https://github.com/anacrolix/btlink
There is currently ArchiveTeam going on