Skip to content

Comment on Sourcehut network outage post-mortem

Comments

I've been using Sourcehut for a couple years now. One thing this outage taught me about the service that I didn't know is that Mercurial (hg) is community maintained:

We also did our best with hg.sr.ht, but it is community maintained

It looks like git.sr.ht is hosted on OVH in France, while hg.sr.ht is hosted on High5! in the Netherlands.

It's not entirely clear to me how this affects their product roadmap or support, but definitely good to know.

This also came as a surprise to me! Not only that but:

restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help

Maintainer, singular!

The only reason i use Sourcehut, and the main reason i pay for it, is because i stubbornly still use Mercurial, and want first-class support for it. With the utmost of respect to M. Chabant, that is not exactly first-class.

With the utmost of respect to M. Chabant, that is not exactly first-class.

It would appear that Ludovic Chabant is working full-time at Epic Games. He is unlikely to have the capacity to be on call for Sourcehut.

I think the complaint was aimed at Sourcehut leaning on a sole volunteer for this service, not at Ludovic Chabant

hg.sr.ht would not exist if not for being community maintained; the cost/benefit ratio does not work out in its favor were we to maintain it internally. The deal is that we provide infrastructure and operations but that the software itself is maintained by the community that needs it. I think this is an advantage of SourceHut's free software model and ethic that allows people to build what they need and to get its infrastructure needs met in a way that wouldn't be possible, for example, on GitHub.

It looks like git.sr.ht is hosted on OVH in France

They explain it here:

However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS

That's such an odd choice for this type of infra. I've had horrendous experiences with OVH in the past and what even worse, terrible customer service. Yes, this was about 8 years ago, and not with France based metal, but still...

Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with an anti-establishment manifesto than any practicality. Then again, I might be wrong.

Well, it's certainly better than their last provider who they couldn't reach during a critical time, and still cannot reasonably communicate with.

They can at least reach and reason with OVH, as mentioned when they got flagged as an out bound DDoS.

Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality

I feel this is a pretty unfair barb considering one of their first moves was reaching out to Cloudflare. Unfortunately, non-http traffic + the need for tls termination on their own servers (pretty sure cloudflare calls this Keyless SSL) squarely lands them as an enterprise customer w/ enterprise pricing.

Drew probably had already entered into agreements with OVH when cloudflare came back around, and we don't have insight on the terms or period for which Cloudflare's second offer was good for.

Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality

Not wanting your traffic MITM'd is anti-establishment. That's where we're at LOL.

hg.sr.ht is operated by SourceHut, but the software is maintained by the community. Ludovic is the primary maintainer and various other Mercurial users participate in its development.

For me there a bit of a language barrier with the terminology. After reading the sentences about hg.sr.ht and community maintenance it seems that some notable meaning is being conveyed about what that means for the operation of the service but its one I'm not smart enough to understand.

I appreciate the service though so I hope the differences between maintained and operated doesn't mean anything in the long term.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.