Hearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their peers that moving git in house won’t result in fewer outages.
In general, paying GH tends to be much cheaper than employing a team of SREs that'll keep your on-prem Git instance up and running 24/7 (esp. including things like backups, disaster recovery, and similar good practices necessary for any enterprise-ready SaaS).
Something tells me that if you can schedule upgrades "outside of business hours", your scale is small (because in a global company, it's always "business hours" somewhere), which makes the cost argument even more prominent.
If you work in a large business that needs git but doesn't have backups and has no disaster recovery plan at all you're going to have much bigger things to worry about than a git repo being offline.
I really like it. I keep source code, 3rd party libraries, images, binary build output and installers. I can easily go back and debug and crash a customer ever sees because I have all the binaries and debugging files.
It depends on your needs and team size. For few tens of dev that need a versioning system with some sugar, a self hosted gitlab can acieve better uptime.
It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up...
Besides the last one, the others you can schedule when developers are not actually working on something, or give a headsup so developers can be prepared in case of errors.
In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
That's because GitHub is used by millions of people every day all over the world. Business hours for them are around the clock. No matter when they update, if it breaks something, many people will hear about it regardless of when the break was deployed.
Yeah, this is exactly my point, their needs don't overlap with your needs, hence sometimes it's better to self-host, because then it's 100% your needs that gets taken care of.
As I said, it depends on your needs. I don't know if you have ever managed a small gitlab self-hosted installation, but it's really trivial, you set up it the first time, and you forget about it. Also, the upgrade process takes few minutes to complete.
Btw, you cannot compare a scheduled maintenance window with an unplanned downtime ... maybe in the middle of an important deployment.
Not add new features every week? Seriously, you cannot tell me that you cannot run a minimal version of a git repo service much more stable than that. I blame the need for new shiny stuff.
Ya'll make me laugh. Don't add any features and "This thing is dead, nothing new in years, find an alternative" But a company actively adds new features their customers are asking for and you're upset that they're adding new features.
And we all know self-hosted software never goes down.
Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
People like to make it sound like running your own server/service is impossible - completely forgetting that's how everything was done not that long ago.
Some wiz-bang new graduates calling themselves Engineers probably haven't a clue - sure, but for everyone else, you probably still have folks employed that did just that on a professional level.
It's not hard - and often it's less expensive. Particularly for a turn-key paid product like Github Enterprise or GitLab.
Comments
Hearts out to all the GitHub SREs who get to fix this, and all the SREs at GitHub’s customers who get to spend the next week convincing management and their peers that moving git in house won’t result in fewer outages.
Our on-prem GH enterprise hasn't gone down in the last year.
Additionally any potentially disruptive work like upgrades etc can be scheduled outside of business hours so that they won't impact users
In general, paying GH tends to be much cheaper than employing a team of SREs that'll keep your on-prem Git instance up and running 24/7 (esp. including things like backups, disaster recovery, and similar good practices necessary for any enterprise-ready SaaS).
Something tells me that if you can schedule upgrades "outside of business hours", your scale is small (because in a global company, it's always "business hours" somewhere), which makes the cost argument even more prominent.
You don't employ a team of SRE's with the sole day-job of git maintenance though, your existing SRE/sysadmin team manages it.
And any large business will already be doing Backups, DR etc for their existing systems.
Any? Are you sure?
Lol don't crush their optimism, the real world will do it for you in due time
Been doing it for 20 years so far.
If you work in a large business that needs git but doesn't have backups and has no disaster recovery plan at all you're going to have much bigger things to worry about than a git repo being offline.
I self-host Subversion and haven’t had an unplanned outage in more than a decade :) (almost two decades?)
How’s your experience with Subversion? What kind of content do store in SVN, BTW?
I really like it. I keep source code, 3rd party libraries, images, binary build output and installers. I can easily go back and debug and crash a customer ever sees because I have all the binaries and debugging files.
It depends on your needs and team size. For few tens of dev that need a versioning system with some sugar, a self hosted gitlab can acieve better uptime.
Gonna call BS on that.
It can achieve better uptime, if you ignore the downtime for upgrades, the downtime for configuration errors, the downtime when the disk fills up...
Besides the last one, the others you can schedule when developers are not actually working on something, or give a headsup so developers can be prepared in case of errors.
In the case of GitHub, Microsoft deploys changes whenever they want, whenever you want it or not.
That's because GitHub is used by millions of people every day all over the world. Business hours for them are around the clock. No matter when they update, if it breaks something, many people will hear about it regardless of when the break was deployed.
Yeah, this is exactly my point, their needs don't overlap with your needs, hence sometimes it's better to self-host, because then it's 100% your needs that gets taken care of.
Good planning and monitoring will also prevent the last point from becoming a problem.
As I said, it depends on your needs. I don't know if you have ever managed a small gitlab self-hosted installation, but it's really trivial, you set up it the first time, and you forget about it. Also, the upgrade process takes few minutes to complete.
Btw, you cannot compare a scheduled maintenance window with an unplanned downtime ... maybe in the middle of an important deployment.
Bitbucket Server
Most of the upgrades happen outside core work hours, which is easy to manage for smaller companies.
So 10 minutes a month for upgrades, zero time for the rest because you are managing it just fine.
so true. At this moment everyone's like, well! This is UNACCEPTABLE. We'll just, we'll just... what, what are you going to do better than github?
Not add new features every week? Seriously, you cannot tell me that you cannot run a minimal version of a git repo service much more stable than that. I blame the need for new shiny stuff.
Ya'll make me laugh. Don't add any features and "This thing is dead, nothing new in years, find an alternative" But a company actively adds new features their customers are asking for and you're upset that they're adding new features.
Pick a lane.
There's a middle ground between "nothing new in years" and "new feature or changes every week".
Curiosity: Not everyone in Hacker News has the same opinions.
Yet in every thread on HN when a service has an outage, the same opinion is spewed.
I know the comments of these posts before I even open them. HN is an echo chamber of predictability.
That you keep opening the discussion page and even participate might say more about you than HN :)
We self-host Gitlab and haven't had an unplanned outage in a year since I joined.
I always wondered if some % of GitHub orgs move to Gitlab when this happens.
I'd be interested to see if there is any correlation there.
Yeah, because Gitlab would never go down.
https://hn.algolia.com/?q=gitlab+down
Gitlab can be self hosted
And we all know self-hosted software never goes down.
Although I guess the benefit is that the CTO gets a psychological lift of looking over Jeremy from IT's shoulder when he tries to fix the issue rather than being at the mercy of some engineers which the company can't fire.
People like to make it sound like running your own server/service is impossible - completely forgetting that's how everything was done not that long ago.
Some wiz-bang new graduates calling themselves Engineers probably haven't a clue - sure, but for everyone else, you probably still have folks employed that did just that on a professional level.
It's not hard - and often it's less expensive. Particularly for a turn-key paid product like Github Enterprise or GitLab.
I've been running self-hosted software, and GitLab in particularly, for over a decade at various jobs.
It almost never goes down. It fact, its up way way way more often that GitHub!
as can Github: https://docs.github.com/en/enterprise-server@3.5/admin/overv...
there's always gitlab