That's a bad conclusion to make: it's the classic tradeoff between importance and urgency. If you only ever do the urgent stuff, the hair-on-fire-has-to-be-done-yesterday stuff, you'll never make time to get to long-term strategic projects.
The payoff for something like unit testing, automated deployment, and continuous integration are over the very, very long haul. If your cost/benefit analysis is always looking at 3- or 6-months out, it'll never seem like a win.
So if you only ever do the urgent stuff and never anything strategic, 3 years later you'll realize that if you'd just sucked it up back at the start and done that stuff, even if meant putting off otherwise urgent features, you'd be further ahead than you are now, because it would have more than made up for the initial investment.
So it's not that you don't do those things because they're not important, but rather because they're never urgent, and because most people's time horizons, especially in a startup, are fairly short.
Still, in 3 years the company might already be bankrupt, and nobody would care about tests anymore. Bankrupt is maybe too extreme, but the particular code module you spent 3 months writing tests for might be replaced by some open source solution or just not being needed anymore.
I kind of see your point, but I find it difficult to deduce a binding rule from all of this. Sometimes it is important to have tests, sometimes other things are more important. You still have to decide on an individual basis.
Thing is, the TDD and technical debt evangelists are typically consultants. Consultants usually earn more money the longer a project takes, and their income is not tied to the yields of the project. Just something to take into consideration imo.
> Consultants usually earn more money the longer a project takes, and their income is not tied to the yields of the project.
Well... that seems oversimplification. When I consult, I ask the price roughly based on the estimation of how much time I'm going to take, but ultimately it is the matter that how much value the client sees in the outcome. Note that the expected outcome and the price are agreed beforehand---if I bump into an unforeseen obstacle and take twice time as my estimation, I can't ask them twice price; basically the more I work, the less I make per hour.
So (1) I have a very strong incentive to make sure my part works in time, and (2) There's less incentive to finish the work earlier than the initial estimation. Thus I tend to put effort to write tests and debugging aids within the time frame (or, in other words, I try to negotiate initial time frame including those tests).
Certainly there's a different pressure on employees; they may adjust priorities and time frames more frequently.
So, it is true that consultants make more money the longer the project takes, but the incentive is in making sure it won't take longer than expected, rather than making it take longer deliberately. (After all, if the consultant is doing ok, there are projects in waiting list so it's not much point to taking one project longer than necessary.)
I agree that there's no overarching rule that makes the tradeoffs easy to analyze.
I will, however, say that after working on the same code base for over 7 years now, and watching the company grow from 15 employees to 400, I can't imagine ever working at a place that didn't have a large investment in unit tests, tech debt elimination, automation, etc. Without that stuff, our products almost certainly would have collapsed under their own weight by now, and our ability to ship predictably and on-time would be gone. Even within the company, we have some groups that have done better than others as far as automating tests (both because of team personality and because of technical issues that make certain types of features harder to test), and it's quite obvious that the groups with the best testing are the groups that are able to make much more predictable progress and that are able to ship on time. The groups with less-good testing tend to be prone to fairly massive schedule slippages due to a ton of late-stage regressions that only get caught when they ship their code out to their internal users.
Once things get to the point where no one person can reasonably understand the full implications of their changes, because the system is just too big and complicated, if you don't have unit tests you're in big, big trouble, and you need to reduce tech debt so you can keep things as comprehensible as possible. Even then, of course, you have to constantly decide how much to invest in testing and infrastructure and cleanup versus how much you invest in forward progress, and there's never an obvious equation that will give you a right answer.
If your code base and team are small and likely to stay that way, such that you can still mercilessly refactor and change the code without introducing a bunch of hidden bugs, then testing doesn't matter as much. If you ever expect the code to get to the point where that becomes less true, and where the possibility of introducing errors increases, then it starts to matter a whole lot more.
Hypothetically, let's assume we built the same product with two teams, one that did a bunch of unit testing and one that didn't (call them Team A and Team B). From my experience, what essentially happens is that Team A ships version 1 first, ships version 2 first but takes about as long to build version 2 as Team B, ships version 3 about the same time (since it takes them longer to build it), experiences a massive schedule slip in version 4 (since the complexity catches up to them and things becomes buggy and they start playing whack-a-mole with bugs), and don't really ever ship a version 5 because their code has so much tech debt that no one can change anything safely without breaking something else unintentionally, and they start contemplating a complete rewrite of the code base. Again, totally contrived situation (it doesn't have to go that way, Team B could still totally screw things up anyway, etc.), but that's roughly what I've seen happen, both at my company and at others.
I don't think it's fair to say that consultants push TDD and tech debt reduction because that means the project will take longer: that's a bit overly cynical. Many, many organizations use unit testing and such in house because it has a huge long-term benefit (as well as generally more predictability in the short term, which is often more valuable than absolute speed), not because some consultant told them to do it.
I am not actually against unit tests, but I have seen it being taken to unhealthy extremes. For example at some companies there are automated tools that check that every method has a unit test. In the end people write unit tests for Java getter and setters and so on. Mind numbing as that task is, people also end up writing bad unit tests just to silence the tool.
A lot of unit tests make sense, but I suspect they also offer plenty of opportunities for idling time away.
No question, it's a fine line . . . you need to be pragmatic and ask "is writing and maintaining this test going to save me more time than it costs?" Over several years, the maintenance of the tests themselves becomes a huge cost, which is something the TDD guys don't seem to talk about much. (My turn for an overly-cynical guess: since many of them are consultants as you've pointed out, they don't hang around with the same code and the same tests for 7 years, so they don't necessarily see how it really plays out). "Bad tests" are actually a huge net negative for development.
If the test is testing something (like a getter or setter) with basically no chance of breaking, then it's a waste of time. If the test is likely to be fragile or non-deterministic, it's a waste of time. If the test is just too hard to write, and it's not too hard to just test by hand, then automating it is probably a waste of time and you should just QA it by hand every so often.
Finding the right balance tends to come back to the old experience and skill thing: you need to have some intuition about which tests will give you the most value (because that part needs to be rock-solid, or because it's hard to get right, or because it's high-change) and which tests need to be thrown away or never written because they aren't worth it.
Taking any development process too far tends to work out poorly, and taking any metric (like test coverage) too seriously is always a bad idea. That said, I've rarely seen unit testing taken way too far; not testing enough and ending up with buggy, regression-riddled software is a far more common failure mode.
This is why it might be an idea to have unit tests AND QA. Be pragmatic with the unit tests and center them around core functionality and things that are hard to test (think very hard about race conditions for example). QA if they're any good should catch the boneheaded exceptions (such as a mis-behaving getter that calls itself).
Comments
That's a bad conclusion to make: it's the classic tradeoff between importance and urgency. If you only ever do the urgent stuff, the hair-on-fire-has-to-be-done-yesterday stuff, you'll never make time to get to long-term strategic projects.
The payoff for something like unit testing, automated deployment, and continuous integration are over the very, very long haul. If your cost/benefit analysis is always looking at 3- or 6-months out, it'll never seem like a win.
So if you only ever do the urgent stuff and never anything strategic, 3 years later you'll realize that if you'd just sucked it up back at the start and done that stuff, even if meant putting off otherwise urgent features, you'd be further ahead than you are now, because it would have more than made up for the initial investment.
So it's not that you don't do those things because they're not important, but rather because they're never urgent, and because most people's time horizons, especially in a startup, are fairly short.
Still, in 3 years the company might already be bankrupt, and nobody would care about tests anymore. Bankrupt is maybe too extreme, but the particular code module you spent 3 months writing tests for might be replaced by some open source solution or just not being needed anymore.
I kind of see your point, but I find it difficult to deduce a binding rule from all of this. Sometimes it is important to have tests, sometimes other things are more important. You still have to decide on an individual basis.
Thing is, the TDD and technical debt evangelists are typically consultants. Consultants usually earn more money the longer a project takes, and their income is not tied to the yields of the project. Just something to take into consideration imo.
> Consultants usually earn more money the longer a project takes, and their income is not tied to the yields of the project.
Well... that seems oversimplification. When I consult, I ask the price roughly based on the estimation of how much time I'm going to take, but ultimately it is the matter that how much value the client sees in the outcome. Note that the expected outcome and the price are agreed beforehand---if I bump into an unforeseen obstacle and take twice time as my estimation, I can't ask them twice price; basically the more I work, the less I make per hour.
So (1) I have a very strong incentive to make sure my part works in time, and (2) There's less incentive to finish the work earlier than the initial estimation. Thus I tend to put effort to write tests and debugging aids within the time frame (or, in other words, I try to negotiate initial time frame including those tests).
Certainly there's a different pressure on employees; they may adjust priorities and time frames more frequently.
So, it is true that consultants make more money the longer the project takes, but the incentive is in making sure it won't take longer than expected, rather than making it take longer deliberately. (After all, if the consultant is doing ok, there are projects in waiting list so it's not much point to taking one project longer than necessary.)
I agree that there's no overarching rule that makes the tradeoffs easy to analyze.
I will, however, say that after working on the same code base for over 7 years now, and watching the company grow from 15 employees to 400, I can't imagine ever working at a place that didn't have a large investment in unit tests, tech debt elimination, automation, etc. Without that stuff, our products almost certainly would have collapsed under their own weight by now, and our ability to ship predictably and on-time would be gone. Even within the company, we have some groups that have done better than others as far as automating tests (both because of team personality and because of technical issues that make certain types of features harder to test), and it's quite obvious that the groups with the best testing are the groups that are able to make much more predictable progress and that are able to ship on time. The groups with less-good testing tend to be prone to fairly massive schedule slippages due to a ton of late-stage regressions that only get caught when they ship their code out to their internal users.
Once things get to the point where no one person can reasonably understand the full implications of their changes, because the system is just too big and complicated, if you don't have unit tests you're in big, big trouble, and you need to reduce tech debt so you can keep things as comprehensible as possible. Even then, of course, you have to constantly decide how much to invest in testing and infrastructure and cleanup versus how much you invest in forward progress, and there's never an obvious equation that will give you a right answer.
If your code base and team are small and likely to stay that way, such that you can still mercilessly refactor and change the code without introducing a bunch of hidden bugs, then testing doesn't matter as much. If you ever expect the code to get to the point where that becomes less true, and where the possibility of introducing errors increases, then it starts to matter a whole lot more.
Hypothetically, let's assume we built the same product with two teams, one that did a bunch of unit testing and one that didn't (call them Team A and Team B). From my experience, what essentially happens is that Team A ships version 1 first, ships version 2 first but takes about as long to build version 2 as Team B, ships version 3 about the same time (since it takes them longer to build it), experiences a massive schedule slip in version 4 (since the complexity catches up to them and things becomes buggy and they start playing whack-a-mole with bugs), and don't really ever ship a version 5 because their code has so much tech debt that no one can change anything safely without breaking something else unintentionally, and they start contemplating a complete rewrite of the code base. Again, totally contrived situation (it doesn't have to go that way, Team B could still totally screw things up anyway, etc.), but that's roughly what I've seen happen, both at my company and at others.
I don't think it's fair to say that consultants push TDD and tech debt reduction because that means the project will take longer: that's a bit overly cynical. Many, many organizations use unit testing and such in house because it has a huge long-term benefit (as well as generally more predictability in the short term, which is often more valuable than absolute speed), not because some consultant told them to do it.
I am not actually against unit tests, but I have seen it being taken to unhealthy extremes. For example at some companies there are automated tools that check that every method has a unit test. In the end people write unit tests for Java getter and setters and so on. Mind numbing as that task is, people also end up writing bad unit tests just to silence the tool.
A lot of unit tests make sense, but I suspect they also offer plenty of opportunities for idling time away.
No question, it's a fine line . . . you need to be pragmatic and ask "is writing and maintaining this test going to save me more time than it costs?" Over several years, the maintenance of the tests themselves becomes a huge cost, which is something the TDD guys don't seem to talk about much. (My turn for an overly-cynical guess: since many of them are consultants as you've pointed out, they don't hang around with the same code and the same tests for 7 years, so they don't necessarily see how it really plays out). "Bad tests" are actually a huge net negative for development.
If the test is testing something (like a getter or setter) with basically no chance of breaking, then it's a waste of time. If the test is likely to be fragile or non-deterministic, it's a waste of time. If the test is just too hard to write, and it's not too hard to just test by hand, then automating it is probably a waste of time and you should just QA it by hand every so often.
Finding the right balance tends to come back to the old experience and skill thing: you need to have some intuition about which tests will give you the most value (because that part needs to be rock-solid, or because it's hard to get right, or because it's high-change) and which tests need to be thrown away or never written because they aren't worth it.
Taking any development process too far tends to work out poorly, and taking any metric (like test coverage) too seriously is always a bad idea. That said, I've rarely seen unit testing taken way too far; not testing enough and ending up with buggy, regression-riddled software is a far more common failure mode.
This is why it might be an idea to have unit tests AND QA. Be pragmatic with the unit tests and center them around core functionality and things that are hard to test (think very hard about race conditions for example). QA if they're any good should catch the boneheaded exceptions (such as a mis-behaving getter that calls itself).
The difficulty is that you always know what's urgent but you cannot be so sure about what has longer term importance.
In the case of unit testing, though, long term is short term enough for me to recognise it as urgent.