Skip to content

Comment on Show HN: Nebula – A network agnostic DHT crawlerparent

Comments

A DHT is a decentralized key-value database, it's most famous use being in the bittorrent protocols, it uses a routing algorithm to guarantee that you can find the peers that can retrieve the value of a known key, granted that you at least know one peer in the network (even if that peer doesn't know the value). Essentially the network is split into buckets and it guarantees that you'll either will be already connected to a peer that knows the value for the key, or that that peer knows a peer whose bucket is closer to the key, you can then recursively ask for peers that are closer and closer until you find one that knows the key, as you do this search you keep track of the peers so the next time you ask for another key you're more likely to know a peer that is closer to it. A typical DHT implementation has you keep track of hundreds of peers to guarantee the robustness of the network.

One issue is that peers go offline and online all the time, so the network is ever changing, if you turn off your client for a week and then come back, your only hope is that at least one of the peers you know is still online, if that's the case then that's fine, if that's not the case, or you're starting the client for the first time, then there's no way for you to connect to the network and query for keys. In bittorrent this is not an issue as most torrents include trackers, the original centralized way of finding peers on the network, but it seems that each project listed on this page has it's own separate DHT network that doesn't connect to the main network (the one used by bittorrent), so for you to connect to these networks for the first time you need to use a bootstrap peer, this is just a normal peer on the network that is known to be always online, usually hosted by owner of the project, and it'll give you a starting point to find other peers in the network.

What this project does in essence, is connect to a bootstrap peer, then use the properties of the routing algorithm to efficiently find out all the peers that are currently online.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.