This is wonderful and shows the problem very clearly.
If your system doesn't have a way to shed load, it will eventually overload.
The problem for many is that when it does finally shed load, the load shedder gets blamed (and turned off). See how many people look for ways to turn off the OOMKiller and how few look to figure out how to get more RAM.
All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more!
I work in a custom fabrication environment, and this solution doesn't really apply, because what happens in a dynamic system is that the bottleneck shifts all around the shop dynamically. It's never just one operation that is the constant bottleneck.
How doesn't it? I've lived in a world of coordinated microservices before, where a nice linear progression of queues only happens at the atomic level; when you zoom out you've got a digraph full of cycles, nodes that come & go in response to various events & schedules, and the transitory nature of those services entering could cause a smooth-sailing pipeline to suddenly seize up because some downstream dependency is servicing higher-priority requests.
The only way we could stay sane was aggressively following Ferd's rule – nobody got to surface an API unless it surfaced HTTP 429s on overload[1], and if you called a downstream API without checking for and responding to a 429, tough noogies, your requests got dropped and you had to figure out what to do to fix it.
Funny enough, at the time I did software for a manufacturing facility that did high-throughput manufacturing for many varying products. I was lucky enough to do a "gemba walk" where they made us engineers put on hard-hats and eye protection and reminded us who actually made the company money, and by god those guys were doing exactly what we did: they had JIT delivery of raw materials / sub-components to assembly lines, and the delivery of resources to the line was always constrained by a fixed buffer, quite literally "how many units you could put on a shelf". You couldn't just toss more product at a line that wasn't ready for it, it would literally not fit.
Sure, they used different terms than we did – muda and muri and value-stream mapping – but the outcome was very similar. Their version of the NOC was a team of managers who watched the cycle time of each line, how long it took material to be transported, which areas were bottlenecking and which were at risk of running into resource starvation, and they had the ability to balance the hot spots by bursting product in, caching upstream components into hold docks, releasing held work product to downstream lines, or even turning on a flex line to work around longer-term constraints. They rarely had fixed, static bottlenecks, because they optimized the life out of them. Their bottlenecks arose when a shipment didn't arrive (and you better believe those were tracked just as aggressively), a machine broke down, an operator got sick, things like that.
But at the end of the day? Same principle: downstream components would only accept as much as they could process from upstream components; you asked before you pushed.
[1]: and if you said "I swear, you can't overload this service, we don't need to send 429s" a swarm of senior engineers descended upon you and only left until you convinced them they were wrong, or – way more likely – they convinced you.
Manufacturing and fabrication are two different worlds. Manufacturing is all about reducing/eliminating variance, whereas fabrication (high-mix, low volume) is all about absorbing variance. Most of the things we build have never been built before, and most of them will never be built again. So the estimates on how long it will take to build one is just that: an estimate. And the process of fabricating it is affected by MANY variables, but I'll just list a few. Materials with long lead times. Materials not arriving on time. Wrong materials sent. A pareto distribution of time it will take to finish an operation depending on which employee I give it to. What else is in the queue for an operation. Whether customers put a hold on the part. And so on, and so on.
Every manufacturing person I've met thinks we're just idiots. Then they come to our shop and it takes them about 1-2 years of trying to "we just need to implement Lean, guys!" before they realize that fabrication has very little, if anything, in common with manufacturing and the only lean principle that really applies is keeping your work area tidy and your tools organized.
Our business model is built around our ability to absorb and manage variance, rather than eliminate it.
I agree that software and fabrication have much more overlap with regard to challenges faced than software and manufacturing. Unfortunately, manufacturing gets most of the academic attention, so job shop scheduling has basically stagnated since the 70s, when academia started becoming aware of Toyota's innovations in manufacturing. Some work has been done using genetic algorithms to try to tackle the subject, but nothing substantive has taken hold in the industry.
That being said, I've found a few resources that help you start to get a handle on fabrication (and software development) and how to manage it. Theory of Constraints by Goldblatt, and it's more executive-friendly companion volume called The Goal are a good place to start, but lacking on detail.
Understanding Variation by Donald J. Wheeler has proven to be a very useful resource.
It's About Time: The Competitive Advantage of Quick Response Manufacturing by Rajan Suri is the executive companion to the book Quick Response Manufacturing: a Compan-Wide Approach to Reducing Lead Times by the same author. Mr. Suri uses a system-dynamics approach to manage capacity and to manage batch sizing, which his research revealed as the keys to optimal lead time reductions, and thereby overall throughput.
The other interesting thing we've found by analyzing the data of our company is that the time it takes to ship a particular piece (measured in days) falls into a long-fat-tail distribution regardless of the scale we analyze. In other words, if we look at one employee's output, or one team's output, or one shop's output, or the company as a whole, the time to ship is in a long-fat-tail distribution.
Because of this, we basically manage by how long something has been in a particular department. The longer it's been sitting, the higher its priority becomes. What we're trying to do is basically manage the exceptions and let the system manage those that are in the normal bell curve.
Hope this helps. Feel free to ask further questions. I find the topic fascinating and not many people share this interest.
After I finished my comment, a thought occurred to me that I wanted to share. One big difference between fabrication and (most) software projects is scope creep. We manage scope quite fiercely, and the weapon we wield is the change order. We are in the fortunate contractual position of being able to charge our customers when they change/add scope.
One of the biggest challenges I've seen (as someone outside the industry, but familiar with it) in software development is feature/scope creep. That seems to really hobble a lot of projects. Star Citizen comes immediately to mind as an outlying, but illustrative example.
We don't really have a lot of feature/scope creep in most of our projects. We do, however, have shared resources between simultaneous projects, which poses a similar challenge.
Yes, we have much more in common with software development than we do with manufacturing. We have more stages of production than software, and we have a challenging situation of shared resources between projects (we are generally working anywhere from 25-50 projects at once in our shop using shared resources), but the variance factor of 1) not knowing exactly how long it will take to make something because it's never been made before, and 2) different employees taking vastly different amounts of time to accomplish the same task are both shared challenges that we face along with software development.
Anything made from steel. Everything from small machined parts all the way up to the swing arms on electric mining shovels. We have some parts we build that require two 60 ton cranes to lift. Everything in between, mostly for the energy industry.
Comments
Here's another article about the same issue I think https://ferd.ca/queues-don-t-fix-overload.html .
Solution: "Step 1. Identify the bottleneck. Step 2: ask the bottleneck for permission to pile more data in"
This is wonderful and shows the problem very clearly.
If your system doesn't have a way to shed load, it will eventually overload.
The problem for many is that when it does finally shed load, the load shedder gets blamed (and turned off). See how many people look for ways to turn off the OOMKiller and how few look to figure out how to get more RAM.
This is also wonderful (emphasis mine):
This is a fantastic article, thank you for sharing!
I work in a custom fabrication environment, and this solution doesn't really apply, because what happens in a dynamic system is that the bottleneck shifts all around the shop dynamically. It's never just one operation that is the constant bottleneck.
How doesn't it? I've lived in a world of coordinated microservices before, where a nice linear progression of queues only happens at the atomic level; when you zoom out you've got a digraph full of cycles, nodes that come & go in response to various events & schedules, and the transitory nature of those services entering could cause a smooth-sailing pipeline to suddenly seize up because some downstream dependency is servicing higher-priority requests.
The only way we could stay sane was aggressively following Ferd's rule – nobody got to surface an API unless it surfaced HTTP 429s on overload[1], and if you called a downstream API without checking for and responding to a 429, tough noogies, your requests got dropped and you had to figure out what to do to fix it.
Funny enough, at the time I did software for a manufacturing facility that did high-throughput manufacturing for many varying products. I was lucky enough to do a "gemba walk" where they made us engineers put on hard-hats and eye protection and reminded us who actually made the company money, and by god those guys were doing exactly what we did: they had JIT delivery of raw materials / sub-components to assembly lines, and the delivery of resources to the line was always constrained by a fixed buffer, quite literally "how many units you could put on a shelf". You couldn't just toss more product at a line that wasn't ready for it, it would literally not fit.
Sure, they used different terms than we did – muda and muri and value-stream mapping – but the outcome was very similar. Their version of the NOC was a team of managers who watched the cycle time of each line, how long it took material to be transported, which areas were bottlenecking and which were at risk of running into resource starvation, and they had the ability to balance the hot spots by bursting product in, caching upstream components into hold docks, releasing held work product to downstream lines, or even turning on a flex line to work around longer-term constraints. They rarely had fixed, static bottlenecks, because they optimized the life out of them. Their bottlenecks arose when a shipment didn't arrive (and you better believe those were tracked just as aggressively), a machine broke down, an operator got sick, things like that.
But at the end of the day? Same principle: downstream components would only accept as much as they could process from upstream components; you asked before you pushed.
[1]: and if you said "I swear, you can't overload this service, we don't need to send 429s" a swarm of senior engineers descended upon you and only left until you convinced them they were wrong, or – way more likely – they convinced you.
Manufacturing and fabrication are two different worlds. Manufacturing is all about reducing/eliminating variance, whereas fabrication (high-mix, low volume) is all about absorbing variance. Most of the things we build have never been built before, and most of them will never be built again. So the estimates on how long it will take to build one is just that: an estimate. And the process of fabricating it is affected by MANY variables, but I'll just list a few. Materials with long lead times. Materials not arriving on time. Wrong materials sent. A pareto distribution of time it will take to finish an operation depending on which employee I give it to. What else is in the queue for an operation. Whether customers put a hold on the part. And so on, and so on.
Every manufacturing person I've met thinks we're just idiots. Then they come to our shop and it takes them about 1-2 years of trying to "we just need to implement Lean, guys!" before they realize that fabrication has very little, if anything, in common with manufacturing and the only lean principle that really applies is keeping your work area tidy and your tools organized.
Our business model is built around our ability to absorb and manage variance, rather than eliminate it.
Interesting, maybe we should drop the manufacturing metaphors in software? Is there any good reading about fabrication?
I agree that software and fabrication have much more overlap with regard to challenges faced than software and manufacturing. Unfortunately, manufacturing gets most of the academic attention, so job shop scheduling has basically stagnated since the 70s, when academia started becoming aware of Toyota's innovations in manufacturing. Some work has been done using genetic algorithms to try to tackle the subject, but nothing substantive has taken hold in the industry.
That being said, I've found a few resources that help you start to get a handle on fabrication (and software development) and how to manage it. Theory of Constraints by Goldblatt, and it's more executive-friendly companion volume called The Goal are a good place to start, but lacking on detail.
Understanding Variation by Donald J. Wheeler has proven to be a very useful resource.
It's About Time: The Competitive Advantage of Quick Response Manufacturing by Rajan Suri is the executive companion to the book Quick Response Manufacturing: a Compan-Wide Approach to Reducing Lead Times by the same author. Mr. Suri uses a system-dynamics approach to manage capacity and to manage batch sizing, which his research revealed as the keys to optimal lead time reductions, and thereby overall throughput.
The other interesting thing we've found by analyzing the data of our company is that the time it takes to ship a particular piece (measured in days) falls into a long-fat-tail distribution regardless of the scale we analyze. In other words, if we look at one employee's output, or one team's output, or one shop's output, or the company as a whole, the time to ship is in a long-fat-tail distribution.
Because of this, we basically manage by how long something has been in a particular department. The longer it's been sitting, the higher its priority becomes. What we're trying to do is basically manage the exceptions and let the system manage those that are in the normal bell curve.
Hope this helps. Feel free to ask further questions. I find the topic fascinating and not many people share this interest.
After I finished my comment, a thought occurred to me that I wanted to share. One big difference between fabrication and (most) software projects is scope creep. We manage scope quite fiercely, and the weapon we wield is the change order. We are in the fortunate contractual position of being able to charge our customers when they change/add scope.
One of the biggest challenges I've seen (as someone outside the industry, but familiar with it) in software development is feature/scope creep. That seems to really hobble a lot of projects. Star Citizen comes immediately to mind as an outlying, but illustrative example.
We don't really have a lot of feature/scope creep in most of our projects. We do, however, have shared resources between simultaneous projects, which poses a similar challenge.
Would this apply to "Software Fabrication" as well?
Yes, we have much more in common with software development than we do with manufacturing. We have more stages of production than software, and we have a challenging situation of shared resources between projects (we are generally working anywhere from 25-50 projects at once in our shop using shared resources), but the variance factor of 1) not knowing exactly how long it will take to make something because it's never been made before, and 2) different employees taking vastly different amounts of time to accomplish the same task are both shared challenges that we face along with software development.
Can you share what kind of products you are in fact fabricating?
Anything made from steel. Everything from small machined parts all the way up to the swing arms on electric mining shovels. We have some parts we build that require two 60 ton cranes to lift. Everything in between, mostly for the energy industry.
Very interesting - thanks!
Love that solution. It's plainly unfair but in my experience so critical to getting things done in business, even setting aside engineering.