In my experience, the moment that creates the greatest amount of legacy code is when you have a team who's project grows large enough that the team splits. At that point, many modules and projects are divided, but anything not explicitly considered ends up as untouchable spaghetti. Most of the time, it's because the code is working, so why put in the work to split it up, just because of a team change? Of course, it stays working, and if needed, you can get the original gang back together to fix it. Until it's been 2 years, and half of them quit, and the others don't remember it quite well enough...
Considering organizational change as the primary driver of system structure change seems like a powerful way of thinking.
"Change your code to match your organization." - Bicking
Some organizations are toxic, though. If they don't change they will ship toxic code or wreck the people shipping the code with the toxicity. I have a friend who works as an engineering management guru. He calls himself a Corporate Oncologist. Many times he finds that the people who hired him are the source of the problem. So he tries to get them to change their ways gently. But sometimes he just throws in the towel before he even begins. "Sorry. I can't help you."
"Change your organization or change your organization" - Jeffries
This is so true. My current place is one where the organization has drastically improved, and you can read it in the code. It's so clear can tell you which group of people wrote what without using git blame just by looking at the code.
On the Criteria for Decomposing Systems into Modules suggests we draw boundaries/interfaces where (we think) change is unlikely. Put the parts we think will change within a module, like implementation details.
This becomes pronunced in unit testing, where the tests add yet another dependency on interfaces, this investment making us more reluctant to change them.
If modules follow organizarionsl structure,
what determines the information flows of organizational structure? And could it perhaps be a good model for code?
I think Ian is not seeing the reason software architecture ends up matching organizational structure.
He's right that engineers can imagine architectures that don't reflect org structure. I mean, of course they can.
However, to have that imagined architecture be implemented, you have to go and talk with whoever controls the parts you don't control. Even if they are totally happy and willing and able to make the changes you want, this step of communicating with them, defining the new cross-team responsibilities, defining interfaces, all the tasks around actually doing this cross team work, come at a much higher cost than intra-team work. Within a team you have more communication bandwidth, you know the people and how decisions are made, and all that means that within your team you can move much more quickly.
Now, people will do the following estimate, whether explicit or implicit to their thinking: is goodness(architecture1) > goodness(architecture2)?
Conway's law comes out of the time locality of that attempt to decide 'what is the best architecture', combined with what is necessarily almost complete uncertainty when trying to estimate future costs and benefits.
Since cost(architecture_internal_only) < cost(architecture_cross_org) whenever the technical advantage of cross_org is less than the added costs of cross organization coordination, there will be many decisions made in favor of internal_only. In fact, whenever there is no clear technical advantage to one or the other, internal_only will always be a better decision.
This is where locality kicks in. Every internal_only decision increases the cost difference in future decisions architecture decisions between internal_only and cross_org, in favor of internal_only.
For example, lets say you are building a web browser. Your team (team A) is working on javascript. Another team (team B) is working on a rendering engine. Both require parsing text based languages, so you might decide to work together to standardize on a common tokenization/lexing engine both teams can use. However, doing that would be quite painful compared to both teams taking the shortest path to solving their own lexing problem. It's not even clear that in the long run it would be better to standardize. So both teams build a Lexer. Now, it so happens that team B has someone with a natural talent for writing Lexers, and their lexer is just plain better. Of course, team A hasn't been sitting around for 2 years.
Team A has a requirement to support the newest version of javascript, and that means some changes to how the parser works. Should team A adopt the team B lexer, which is better in every measurable way, or just update their current parser? Well, the cost of using the team B lexer is now HUGE, since it would mean digging out the current lexer. Maybe the team B parser uses multiple passes, where team A has lexing integrated into the parser. The specifics aren't super important, it would probably be insane to take a 2 week project and turn it into a 3 month project just to avoid the duplication. Not to mention, they have 2 years of getting the current solution working well in the real world. Dozens and dozens of real world crap has been quietly accounted for and dealt with. There aren't any bugs reported against the current solution. Changing is more expensive and will potentially introduce bugs. So now both teams just keep going with their own lexing strategies forever, it will never be the right decision to remove this reflection of the organizational structure.
In year 3 the teams get reshuffled, and one team is now responsible for all the parsing. They inherit both the HTML parser and the JS parser. What do you think? How long will it be before they see how nice it would be to unify the parsers? How would you feel if you were working on both, wouldn't you feel the urge to 'clean this up'?
Now, what if early on, you had decided to in fact do the 'one lexer to rule them all'. You may have discovered in 2 years that the tasks of parsing HTML and JS are just fundamentally so different that it was hopeless to try to share functionality in this way. The JS parsing is hopelessly slow compared to competitors, because the lexer needs to deal with tons of insanity in real world HTML. Or maybe it works out great? It's just too hard to make a reasonable guess up front to justify choosing a more expensive, longer plan.
Comments
I really like this.
In my experience, the moment that creates the greatest amount of legacy code is when you have a team who's project grows large enough that the team splits. At that point, many modules and projects are divided, but anything not explicitly considered ends up as untouchable spaghetti. Most of the time, it's because the code is working, so why put in the work to split it up, just because of a team change? Of course, it stays working, and if needed, you can get the original gang back together to fix it. Until it's been 2 years, and half of them quit, and the others don't remember it quite well enough...
Considering organizational change as the primary driver of system structure change seems like a powerful way of thinking.
The worst parts of the code last the longest. The best parts of the code change all the time.
A great example of the Lindy Effect.[1] Something which took me too long to actually remember the name of...
[1] https://en.wikipedia.org/wiki/Lindy_effect
"Change your code to match your organization." - Bicking
Some organizations are toxic, though. If they don't change they will ship toxic code or wreck the people shipping the code with the toxicity. I have a friend who works as an engineering management guru. He calls himself a Corporate Oncologist. Many times he finds that the people who hired him are the source of the problem. So he tries to get them to change their ways gently. But sometimes he just throws in the towel before he even begins. "Sorry. I can't help you."
"Change your organization or change your organization" - Jeffries
This is so true. My current place is one where the organization has drastically improved, and you can read it in the code. It's so clear can tell you which group of people wrote what without using git blame just by looking at the code.
Sounds like he is advocating the old "reverse Conway maneuver". It works, I recommend it.
On the Criteria for Decomposing Systems into Modules suggests we draw boundaries/interfaces where (we think) change is unlikely. Put the parts we think will change within a module, like implementation details.
This becomes pronunced in unit testing, where the tests add yet another dependency on interfaces, this investment making us more reluctant to change them.
If modules follow organizarionsl structure, what determines the information flows of organizational structure? And could it perhaps be a good model for code?
I think Ian is not seeing the reason software architecture ends up matching organizational structure.
He's right that engineers can imagine architectures that don't reflect org structure. I mean, of course they can.
However, to have that imagined architecture be implemented, you have to go and talk with whoever controls the parts you don't control. Even if they are totally happy and willing and able to make the changes you want, this step of communicating with them, defining the new cross-team responsibilities, defining interfaces, all the tasks around actually doing this cross team work, come at a much higher cost than intra-team work. Within a team you have more communication bandwidth, you know the people and how decisions are made, and all that means that within your team you can move much more quickly.
Now, people will do the following estimate, whether explicit or implicit to their thinking: is goodness(architecture1) > goodness(architecture2)?
Conway's law comes out of the time locality of that attempt to decide 'what is the best architecture', combined with what is necessarily almost complete uncertainty when trying to estimate future costs and benefits.
Since cost(architecture_internal_only) < cost(architecture_cross_org) whenever the technical advantage of cross_org is less than the added costs of cross organization coordination, there will be many decisions made in favor of internal_only. In fact, whenever there is no clear technical advantage to one or the other, internal_only will always be a better decision.
This is where locality kicks in. Every internal_only decision increases the cost difference in future decisions architecture decisions between internal_only and cross_org, in favor of internal_only.
For example, lets say you are building a web browser. Your team (team A) is working on javascript. Another team (team B) is working on a rendering engine. Both require parsing text based languages, so you might decide to work together to standardize on a common tokenization/lexing engine both teams can use. However, doing that would be quite painful compared to both teams taking the shortest path to solving their own lexing problem. It's not even clear that in the long run it would be better to standardize. So both teams build a Lexer. Now, it so happens that team B has someone with a natural talent for writing Lexers, and their lexer is just plain better. Of course, team A hasn't been sitting around for 2 years.
Team A has a requirement to support the newest version of javascript, and that means some changes to how the parser works. Should team A adopt the team B lexer, which is better in every measurable way, or just update their current parser? Well, the cost of using the team B lexer is now HUGE, since it would mean digging out the current lexer. Maybe the team B parser uses multiple passes, where team A has lexing integrated into the parser. The specifics aren't super important, it would probably be insane to take a 2 week project and turn it into a 3 month project just to avoid the duplication. Not to mention, they have 2 years of getting the current solution working well in the real world. Dozens and dozens of real world crap has been quietly accounted for and dealt with. There aren't any bugs reported against the current solution. Changing is more expensive and will potentially introduce bugs. So now both teams just keep going with their own lexing strategies forever, it will never be the right decision to remove this reflection of the organizational structure.
In year 3 the teams get reshuffled, and one team is now responsible for all the parsing. They inherit both the HTML parser and the JS parser. What do you think? How long will it be before they see how nice it would be to unify the parsers? How would you feel if you were working on both, wouldn't you feel the urge to 'clean this up'?
Now, what if early on, you had decided to in fact do the 'one lexer to rule them all'. You may have discovered in 2 years that the tasks of parsing HTML and JS are just fundamentally so different that it was hopeless to try to share functionality in this way. The JS parsing is hopelessly slow compared to competitors, because the lexer needs to deal with tons of insanity in real world HTML. Or maybe it works out great? It's just too hard to make a reasonable guess up front to justify choosing a more expensive, longer plan.
I'm not sure what the author thinks 'corollary' means here. (Something to do with correlation? Not the definition I'm used to, anyway.)
A proposition that is logically derived from one already proved, usually immediately before.