If you're using a cloud running test service like Travis CI pushing to GitHub can trigger your whole deployment process starting with running the unit and integration tests.
Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down?
Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- there had absolutely better be a failsafe that does not involve GitHub and Travis in your architecture.
Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS records, which are hosted on another third party's DNS servers.
Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, you didn't need to do that. If you still did I'd like to know why you did that, because in an era where even the CIA relies on third parties to accomplish mission critical tasks it doesn't seem to make a whole lot of sense.
And that trauma you are talking about - it doesn't happen. Not rarely, not ever. For years. Tonight I only happened to see that GitHub was down because I was looking at HN anyway.
Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies,
The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think.
Based on this comment and the recent reply in this thread I can tell you've mostly worked with small-scale architectures. Third party tooling and workflows do make a lot of sense at a small scale, but the point at which you outgrow those solutions comes a lot sooner than you think. When I arrived at Foursquare the entire operation was on Amazon; when I left a year and a half later, much of it was on physical equipment. And Foursquare is not a Google-scale operation -- virtualization and customer cotenancy just have a serious impact on SLA that is less pronounced at smaller scales.
It's easy to think your experiences represent the industry, as your comments suggest. It's also easy to think HN represents the industry, where startups reign supreme and everybody loves working Lean Devops. The fact is, neither of those statements are accurate, and beyond a six (or maybe seven) figure architecture you start having a harder time justifying third parties financially and operationally.
I do use Amazon currently, just as an off-site backup for on-site monitoring that I've built. That's common.
I recognize that the costs become less straightforward the larger you get, and that a number of larger companies have to develop their own bare metal infrastructure. But I was including companies exactly like Foursquare in that handful of tech companies.
I'd still like to see an actual cost breakdown, because to me hiring a full-time staff of 5 or 10 people to run such an infrastructure, at $80,000/year or more a pop, seems like it could quickly get just as expensive or more expensive than outsourcing those costs to another organization.
There has to be some reason Netflix is able to justify placing their entire high bandwidth streaming operation on AWS, a third party that relies on full virtualization.
Github's uptime is good enough that I don't see anything inherently awful about a workflow that makes development difficult when Github is down, but any production deployment process that strictly requires anything beyond a working internet connection and a working server to deploy to is insane.
Works pretty well for me. The argument you just made could have - and has - been about any other piece of infrastructure that we now work with third parties for. DNS, EMail, managed hosting, VPS hosting, off-site backups, etc, etc etc. I rely on GitHub to store my code and handle code sharing from separated networks behind NAT - why can't I rely on it for code deployment?
The days of isolating yourself from third parties are over be cause it really doesn't make sense to spend a bunch of man hours setting up and maintaining an infrastructure that GitHub - with the rare exception of tonight - specializes in and has an entire staff dedicated to keeping up and running. Third parties have become mission critical because they save money and time and are generally reliable.
If Travis CI or GitHub stop working I can do a number of things if I really need to deploy - for instance, push to production anyway, which, btw, is also hosted on a third party hosting provider. If several large services start falling like dominoes at once we have bigger problems than just whether my service is running.
Comments
If you're using a cloud running test service like Travis CI pushing to GitHub can trigger your whole deployment process starting with running the unit and integration tests.
Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down?
Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- there had absolutely better be a failsafe that does not involve GitHub and Travis in your architecture.
Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS records, which are hosted on another third party's DNS servers.
Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, you didn't need to do that. If you still did I'd like to know why you did that, because in an era where even the CIA relies on third parties to accomplish mission critical tasks it doesn't seem to make a whole lot of sense.
And that trauma you are talking about - it doesn't happen. Not rarely, not ever. For years. Tonight I only happened to see that GitHub was down because I was looking at HN anyway.
The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think.
Based on this comment and the recent reply in this thread I can tell you've mostly worked with small-scale architectures. Third party tooling and workflows do make a lot of sense at a small scale, but the point at which you outgrow those solutions comes a lot sooner than you think. When I arrived at Foursquare the entire operation was on Amazon; when I left a year and a half later, much of it was on physical equipment. And Foursquare is not a Google-scale operation -- virtualization and customer cotenancy just have a serious impact on SLA that is less pronounced at smaller scales.
It's easy to think your experiences represent the industry, as your comments suggest. It's also easy to think HN represents the industry, where startups reign supreme and everybody loves working Lean Devops. The fact is, neither of those statements are accurate, and beyond a six (or maybe seven) figure architecture you start having a harder time justifying third parties financially and operationally.
I do use Amazon currently, just as an off-site backup for on-site monitoring that I've built. That's common.
I recognize that the costs become less straightforward the larger you get, and that a number of larger companies have to develop their own bare metal infrastructure. But I was including companies exactly like Foursquare in that handful of tech companies.
I'd still like to see an actual cost breakdown, because to me hiring a full-time staff of 5 or 10 people to run such an infrastructure, at $80,000/year or more a pop, seems like it could quickly get just as expensive or more expensive than outsourcing those costs to another organization.
There has to be some reason Netflix is able to justify placing their entire high bandwidth streaming operation on AWS, a third party that relies on full virtualization.
Two third parties, even.
Github's uptime is good enough that I don't see anything inherently awful about a workflow that makes development difficult when Github is down, but any production deployment process that strictly requires anything beyond a working internet connection and a working server to deploy to is insane.
Works pretty well for me. The argument you just made could have - and has - been about any other piece of infrastructure that we now work with third parties for. DNS, EMail, managed hosting, VPS hosting, off-site backups, etc, etc etc. I rely on GitHub to store my code and handle code sharing from separated networks behind NAT - why can't I rely on it for code deployment?
The days of isolating yourself from third parties are over be cause it really doesn't make sense to spend a bunch of man hours setting up and maintaining an infrastructure that GitHub - with the rare exception of tonight - specializes in and has an entire staff dedicated to keeping up and running. Third parties have become mission critical because they save money and time and are generally reliable.
If Travis CI or GitHub stop working I can do a number of things if I really need to deploy - for instance, push to production anyway, which, btw, is also hosted on a third party hosting provider. If several large services start falling like dominoes at once we have bigger problems than just whether my service is running.