I'm that guy whom the post is talking about. I decided against weighing in on the original thread on HN because I was still dealing with the downfall. A couple of things that I noticed:
* Our users were very understanding of what happened. We have received nothing but encouragement to keep on working.
* Some comments on HN were nasty. I'm glad to be 33 and not 23. Otherwise, I could have been driven away from building my product because of my own incompetence.
* Many commented on devs vs ops. The way I see it, I can ask a dev to supervise the work of an ops contractor. I can't hire ops + dev at this stage.
Any start-up has three main constraints: time, money and talent. These are not set in stone, you can use time to produce money (consulting), you can use money to buy talent (hiring), and you can even convert time in talent (training).
So, when people say "let professionals handle it". Well, no, my particular set of constraints won't allow me to do this. My budget for this is around $100/mo. In an event where I completely run out of money I'd have to take down the site indefinitely, which causes the same effect as an HD loss.
It is clear now that I lack enough resources to run a complex app reliably. My focus in the next months is procuring those resources (money) so I can put it back into the product (ops and devs).
Might be the difference between startup and company. So while you're proving out your idea you probably don't need a whole lot of ops (but doing backups and source code control is just good development practice) As a company you need 5 things;
1) Design talent - presenting the coolness in a way that others can understand
2) Technical talent - taking great ideas and composing systems to make them real.
3) Legal talent - to keep you covered from the folks who would want to kill you when you are successful.
4) Operations talent - making sure that you can keep doing what your doing over time
5) Sales talent - communicating what you are doing in a way that the picture appears in someone else's head, and the value is clear.
You need coverage on all 5 of those skill sets, you might find someone who can cover two or three (they will be in high demand) or you may need to recruit to fill them, but unless you have them all you're highest risk of failure will be that undefended flank.
Yes, but each of these come at the proper moment. The iTunes Store covers most of 3, 4 and 5 as an indie dev and then, if you feel comfortable, scale up to be a "company".
>So, when people say "let professionals handle it". Well, no, my particular set of constraints won't allow me to do this. My budget for this is around $100/mo. In an event where I completely run out of money I'd have to take down the site indefinitely, which causes the same effect as an HD loss.
well, drat. I was talking about launching a "tested backups" service, but it's just not really worth my time until you get to the $500/month level or so, and I'd probably want a setup fee on top of that.
(For that, I'd give you a full working replication of your production site, hosted on my stuff- something that, in case of emergency, you could cut over your dns and just run with. Something that you could go to at any time and check on by going to yourdomain.backups.prgmr.com or something. Obviously, this would take me setting up some sort of replication of your database. Obviously, this also means that I'd need to know your application well enough to figure out how to make the running backup not conflict with the primary, and how to cut over to the backup /as/ a primary and how to cut back.)
I mean, I can do basic backups really cheaply, but testing them? that... that takes effort. Effort and understanding the application. And untested backups, meh, there's no reason for you to pay me to do it (maybe you pay me for space, but that's the cheap part.) there are thousands of services that will cheaply give you a place to hold files.
Huh. Most of the work, on my end, would be up front. What if I charged you $100/month, but made you pre-pay a year in advance or something? that might be worth it for me. (assuming I had the option to back out and refund your money within the first X days should your application prove to be too difficult to replicate.)
If the production server is running as a VM to begin with, how hard is it to take a snapshot of the whole VM and replicate that? (It does sort of break down if you have a database that's continuously updated and you want to try to have a continuously updated copy of the database, though.)
How hard is it? Well, about as hard as writing bug free code is. You don't need testers, right, just don't put bugs in in the first place! Or is it not easy after all?
well, just taking an image and throwing it somewhere is pretty easy (takes a bit of disk/network bandwidth, but it's not hard, if you have enough of those)
If you are short on disk/network bandwidth, the 'snapshots over the network' capabilities of things like zfs can help, while still remaining fairly simple.
hell, plain old rsync can work pretty well, and helps a lot with network bandwidth, even if it doesn't help so much with disk bandwidth.
And yeah, just taking disk images (or even tarballs) and stashing them somewhere cheap is a good start. much better than nothing.
Now, the downside to untested backups? when you need the data, something always goes wrong and it becomes this huge emergency that you have to deal with right now - and sometimes? even if you do everything right from that point on, the data is just gone.
I mean, that's a fair tradeoff a lot of times; if everything goes well, you've put little effort into backups, but you are still at a point where if something goes wrong, you can work hard and probably get your data back. I mean, yeah, you still have a reasonably high chance of losing everything, but your chances of recovery are way better than if you have no backups, and the effort level (if you don't need the backups) is not that much higher than no backups.
The thing is, the original subject was people who didn't want to deal with backups. "Oh," I thought, "Why don't you pay me to worry about your backups?" If I'm going to accept money from you to manage your backups? well, I have to set some expectations. most people aren't going to want to pay very much for untested backups, 'cause it really isn't that much work, until the excrement intersects the turbines, at which point it's a giant emergency, and a huge amount of work. At that point, I'm in a position where some $10/month customer is expecting me to do a couple grand worth of work; I mean, as a $10/month customer, I have never seriously looked at your app before. I've gotta restore from a backup that was taken without thinking about how your app works and make it work when I don't even know what "working" looks like, so it's going to be a lot of effort. Either I do it for free and I'm unhappy, or I hit them with a huge unexpected bill, and they are unhappy. (well, a $10/month customer, generally speaking, simply isn't going to pay a multi-kilobuck bill, and this is easily several thousand dollars worth of work, so in that case I'm unhappy and they are unhappy.) I try to be careful about setting expectations that I can actually meet, and I try to only accept customers who will be happy with a service that I can actually provide. - So yeah, while I think untested backups are a fine first step, and might even be what makes most economic sense for most people, configuring untested backups is not a service I would be willing to provide. If that's what you want, do it yourself.
However, if I put more effort into a tested recovery procedure up front, and more effort into testing that procedure in a routine, verifiable way ongoing? Sure, a lot more effort is being spent, effort that didn't need to be spent if nothing goes wrong, and I'm going to charge you for that. It will be more expensive. But I'm charging you a predictable amount; you knew what you were getting into when you signed up, so you aren't so unhappy. Then, in the unlikely event of a water landing? It's really not that big of a deal; you've been paying your monthly dues, I know the basics of your system, I've set aside some spare hosting resources, and I feel okay about getting woken up to make sure your restore goes okay, and the restore probably will go okay. I mean, it'd cost more money up-front, but it would set expectations in such a way that those expectations could be met.
My break is over; I need to get back to dealing with one of those giant emergency situations caused by a lack of tested backups... on one of my systems, in fact.
I think $100/mo is a good price for tested backups. But it's only in hindsight that I can see this is a good deal. To the average person, backing up means connecting an external HD and let Time Machine/whatever take it from there.
Comments
I'm that guy whom the post is talking about. I decided against weighing in on the original thread on HN because I was still dealing with the downfall. A couple of things that I noticed:
* Our users were very understanding of what happened. We have received nothing but encouragement to keep on working.
* Some comments on HN were nasty. I'm glad to be 33 and not 23. Otherwise, I could have been driven away from building my product because of my own incompetence.
* Many commented on devs vs ops. The way I see it, I can ask a dev to supervise the work of an ops contractor. I can't hire ops + dev at this stage.
Any start-up has three main constraints: time, money and talent. These are not set in stone, you can use time to produce money (consulting), you can use money to buy talent (hiring), and you can even convert time in talent (training).
So, when people say "let professionals handle it". Well, no, my particular set of constraints won't allow me to do this. My budget for this is around $100/mo. In an event where I completely run out of money I'd have to take down the site indefinitely, which causes the same effect as an HD loss.
It is clear now that I lack enough resources to run a complex app reliably. My focus in the next months is procuring those resources (money) so I can put it back into the product (ops and devs).
Might be the difference between startup and company. So while you're proving out your idea you probably don't need a whole lot of ops (but doing backups and source code control is just good development practice) As a company you need 5 things;
1) Design talent - presenting the coolness in a way that others can understand
2) Technical talent - taking great ideas and composing systems to make them real.
3) Legal talent - to keep you covered from the folks who would want to kill you when you are successful.
4) Operations talent - making sure that you can keep doing what your doing over time
5) Sales talent - communicating what you are doing in a way that the picture appears in someone else's head, and the value is clear.
You need coverage on all 5 of those skill sets, you might find someone who can cover two or three (they will be in high demand) or you may need to recruit to fill them, but unless you have them all you're highest risk of failure will be that undefended flank.
Yes, but each of these come at the proper moment. The iTunes Store covers most of 3, 4 and 5 as an indie dev and then, if you feel comfortable, scale up to be a "company".
>So, when people say "let professionals handle it". Well, no, my particular set of constraints won't allow me to do this. My budget for this is around $100/mo. In an event where I completely run out of money I'd have to take down the site indefinitely, which causes the same effect as an HD loss.
well, drat. I was talking about launching a "tested backups" service, but it's just not really worth my time until you get to the $500/month level or so, and I'd probably want a setup fee on top of that.
(For that, I'd give you a full working replication of your production site, hosted on my stuff- something that, in case of emergency, you could cut over your dns and just run with. Something that you could go to at any time and check on by going to yourdomain.backups.prgmr.com or something. Obviously, this would take me setting up some sort of replication of your database. Obviously, this also means that I'd need to know your application well enough to figure out how to make the running backup not conflict with the primary, and how to cut over to the backup /as/ a primary and how to cut back.)
I mean, I can do basic backups really cheaply, but testing them? that... that takes effort. Effort and understanding the application. And untested backups, meh, there's no reason for you to pay me to do it (maybe you pay me for space, but that's the cheap part.) there are thousands of services that will cheaply give you a place to hold files.
Huh. Most of the work, on my end, would be up front. What if I charged you $100/month, but made you pre-pay a year in advance or something? that might be worth it for me. (assuming I had the option to back out and refund your money within the first X days should your application prove to be too difficult to replicate.)
If the production server is running as a VM to begin with, how hard is it to take a snapshot of the whole VM and replicate that? (It does sort of break down if you have a database that's continuously updated and you want to try to have a continuously updated copy of the database, though.)
How hard is it? Well, about as hard as writing bug free code is. You don't need testers, right, just don't put bugs in in the first place! Or is it not easy after all?
well, just taking an image and throwing it somewhere is pretty easy (takes a bit of disk/network bandwidth, but it's not hard, if you have enough of those)
If you are short on disk/network bandwidth, the 'snapshots over the network' capabilities of things like zfs can help, while still remaining fairly simple.
hell, plain old rsync can work pretty well, and helps a lot with network bandwidth, even if it doesn't help so much with disk bandwidth.
And yeah, just taking disk images (or even tarballs) and stashing them somewhere cheap is a good start. much better than nothing.
Now, the downside to untested backups? when you need the data, something always goes wrong and it becomes this huge emergency that you have to deal with right now - and sometimes? even if you do everything right from that point on, the data is just gone.
I mean, that's a fair tradeoff a lot of times; if everything goes well, you've put little effort into backups, but you are still at a point where if something goes wrong, you can work hard and probably get your data back. I mean, yeah, you still have a reasonably high chance of losing everything, but your chances of recovery are way better than if you have no backups, and the effort level (if you don't need the backups) is not that much higher than no backups.
The thing is, the original subject was people who didn't want to deal with backups. "Oh," I thought, "Why don't you pay me to worry about your backups?" If I'm going to accept money from you to manage your backups? well, I have to set some expectations. most people aren't going to want to pay very much for untested backups, 'cause it really isn't that much work, until the excrement intersects the turbines, at which point it's a giant emergency, and a huge amount of work. At that point, I'm in a position where some $10/month customer is expecting me to do a couple grand worth of work; I mean, as a $10/month customer, I have never seriously looked at your app before. I've gotta restore from a backup that was taken without thinking about how your app works and make it work when I don't even know what "working" looks like, so it's going to be a lot of effort. Either I do it for free and I'm unhappy, or I hit them with a huge unexpected bill, and they are unhappy. (well, a $10/month customer, generally speaking, simply isn't going to pay a multi-kilobuck bill, and this is easily several thousand dollars worth of work, so in that case I'm unhappy and they are unhappy.) I try to be careful about setting expectations that I can actually meet, and I try to only accept customers who will be happy with a service that I can actually provide. - So yeah, while I think untested backups are a fine first step, and might even be what makes most economic sense for most people, configuring untested backups is not a service I would be willing to provide. If that's what you want, do it yourself.
However, if I put more effort into a tested recovery procedure up front, and more effort into testing that procedure in a routine, verifiable way ongoing? Sure, a lot more effort is being spent, effort that didn't need to be spent if nothing goes wrong, and I'm going to charge you for that. It will be more expensive. But I'm charging you a predictable amount; you knew what you were getting into when you signed up, so you aren't so unhappy. Then, in the unlikely event of a water landing? It's really not that big of a deal; you've been paying your monthly dues, I know the basics of your system, I've set aside some spare hosting resources, and I feel okay about getting woken up to make sure your restore goes okay, and the restore probably will go okay. I mean, it'd cost more money up-front, but it would set expectations in such a way that those expectations could be met.
My break is over; I need to get back to dealing with one of those giant emergency situations caused by a lack of tested backups... on one of my systems, in fact.
I think $100/mo is a good price for tested backups. But it's only in hindsight that I can see this is a good deal. To the average person, backing up means connecting an external HD and let Time Machine/whatever take it from there.
You could team up with a technical person, though!