Sure, but given that developer time is a bounded resource, we can't really expect to get both fewer restarts and faster restarts. I sympathize with your example, but I would expect that you had already seriously optimized your startup process, and since systemd doesn't make services start more quickly (it only does a better job of parallelizing them), I doubt that it would have made a huge difference in your case. Assuming even a very generous 5-second startup difference, that would have only made a $350 difference per incident.
> but given that developer time is a bounded resource, we can't really expect to get both fewer restarts and faster restarts.
Your're going to have to clarify, as this statement doesn't make any sense.
You're right though. In this instance, system startup was already highly optimized. In fact, the order of startup was shifted around to get Oracle (the DB in this example) up and operational as early as possible.
On the other hand, with the introduction of SMF, that "need to optimise" largely went away. Sure, SMF didn't make much a difference in this particular instance, but parallelisation does make a difference when you're talking about total system (not service) startup time. Further it can make a difference on system that run multiple services. If 5 services have the same dependency set, then a parallel startup means that services 2-5 become available earlier versus a linear startup.
Finally, shaving 5 seconds off of service startup _does_ make a difference. While the above example means $350, there's a few things to consider.
1) a $350 loss is stil a loss. From a business perspective, a loss, no matter how small, is still to be avoided. There's other considerations as well, such as the impact of the extra 5 seconds on my budget (the SLAs mean that the loss comes out of my budget).
2) In the realm of big enterprise, this is a relatively small outage cost. In fact, I've managed system with an order of magnitude larger failure cost (SLAs are bitch).
3) It's difficult to factor soft costs. Things such as customer confidence, indirect productivity loss, etc. 5 seconds, again, means services are available sooner, lessening the impact of these soft, and difficult to quantify, costs.
I've always architected systems such that restart time didn't matter - the systems were able to fail over. However, in flight operations would fail, which meant that a restart would, in fact, cost >$50k - regardless of the length of time it took for that machine to come back.
Faster restart time is irrelevant to me. I'm already designed to deal with outages. Reducing the impact of a restart is extremely important (making them fewer in number, smaller in scope).
This usually depends on how business critical it is. If some service is not very business critical, then how do you justify (or get budget for) the extra resources you need to make it way more reliable?
But even when not business critical, it is nice that you can bring it up quickly.
Comments
Sure, but given that developer time is a bounded resource, we can't really expect to get both fewer restarts and faster restarts. I sympathize with your example, but I would expect that you had already seriously optimized your startup process, and since systemd doesn't make services start more quickly (it only does a better job of parallelizing them), I doubt that it would have made a huge difference in your case. Assuming even a very generous 5-second startup difference, that would have only made a $350 difference per incident.
> but given that developer time is a bounded resource, we can't really expect to get both fewer restarts and faster restarts.
Your're going to have to clarify, as this statement doesn't make any sense.
You're right though. In this instance, system startup was already highly optimized. In fact, the order of startup was shifted around to get Oracle (the DB in this example) up and operational as early as possible.
On the other hand, with the introduction of SMF, that "need to optimise" largely went away. Sure, SMF didn't make much a difference in this particular instance, but parallelisation does make a difference when you're talking about total system (not service) startup time. Further it can make a difference on system that run multiple services. If 5 services have the same dependency set, then a parallel startup means that services 2-5 become available earlier versus a linear startup.
Finally, shaving 5 seconds off of service startup _does_ make a difference. While the above example means $350, there's a few things to consider.
1) a $350 loss is stil a loss. From a business perspective, a loss, no matter how small, is still to be avoided. There's other considerations as well, such as the impact of the extra 5 seconds on my budget (the SLAs mean that the loss comes out of my budget).
2) In the realm of big enterprise, this is a relatively small outage cost. In fact, I've managed system with an order of magnitude larger failure cost (SLAs are bitch).
3) It's difficult to factor soft costs. Things such as customer confidence, indirect productivity loss, etc. 5 seconds, again, means services are available sooner, lessening the impact of these soft, and difficult to quantify, costs.
I've always architected systems such that restart time didn't matter - the systems were able to fail over. However, in flight operations would fail, which meant that a restart would, in fact, cost >$50k - regardless of the length of time it took for that machine to come back.
Faster restart time is irrelevant to me. I'm already designed to deal with outages. Reducing the impact of a restart is extremely important (making them fewer in number, smaller in scope).
This usually depends on how business critical it is. If some service is not very business critical, then how do you justify (or get budget for) the extra resources you need to make it way more reliable?
But even when not business critical, it is nice that you can bring it up quickly.