I agree that there's no overarching rule that makes the tradeoffs easy to analyze.
I will, however, say that after working on the same code base for over 7 years now, and watching the company grow from 15 employees to 400, I can't imagine ever working at a place that didn't have a large investment in unit tests, tech debt elimination, automation, etc. Without that stuff, our products almost certainly would have collapsed under their own weight by now, and our ability to ship predictably and on-time would be gone. Even within the company, we have some groups that have done better than others as far as automating tests (both because of team personality and because of technical issues that make certain types of features harder to test), and it's quite obvious that the groups with the best testing are the groups that are able to make much more predictable progress and that are able to ship on time. The groups with less-good testing tend to be prone to fairly massive schedule slippages due to a ton of late-stage regressions that only get caught when they ship their code out to their internal users.
Once things get to the point where no one person can reasonably understand the full implications of their changes, because the system is just too big and complicated, if you don't have unit tests you're in big, big trouble, and you need to reduce tech debt so you can keep things as comprehensible as possible. Even then, of course, you have to constantly decide how much to invest in testing and infrastructure and cleanup versus how much you invest in forward progress, and there's never an obvious equation that will give you a right answer.
If your code base and team are small and likely to stay that way, such that you can still mercilessly refactor and change the code without introducing a bunch of hidden bugs, then testing doesn't matter as much. If you ever expect the code to get to the point where that becomes less true, and where the possibility of introducing errors increases, then it starts to matter a whole lot more.
Hypothetically, let's assume we built the same product with two teams, one that did a bunch of unit testing and one that didn't (call them Team A and Team B). From my experience, what essentially happens is that Team A ships version 1 first, ships version 2 first but takes about as long to build version 2 as Team B, ships version 3 about the same time (since it takes them longer to build it), experiences a massive schedule slip in version 4 (since the complexity catches up to them and things becomes buggy and they start playing whack-a-mole with bugs), and don't really ever ship a version 5 because their code has so much tech debt that no one can change anything safely without breaking something else unintentionally, and they start contemplating a complete rewrite of the code base. Again, totally contrived situation (it doesn't have to go that way, Team B could still totally screw things up anyway, etc.), but that's roughly what I've seen happen, both at my company and at others.
I don't think it's fair to say that consultants push TDD and tech debt reduction because that means the project will take longer: that's a bit overly cynical. Many, many organizations use unit testing and such in house because it has a huge long-term benefit (as well as generally more predictability in the short term, which is often more valuable than absolute speed), not because some consultant told them to do it.
I am not actually against unit tests, but I have seen it being taken to unhealthy extremes. For example at some companies there are automated tools that check that every method has a unit test. In the end people write unit tests for Java getter and setters and so on. Mind numbing as that task is, people also end up writing bad unit tests just to silence the tool.
A lot of unit tests make sense, but I suspect they also offer plenty of opportunities for idling time away.
No question, it's a fine line . . . you need to be pragmatic and ask "is writing and maintaining this test going to save me more time than it costs?" Over several years, the maintenance of the tests themselves becomes a huge cost, which is something the TDD guys don't seem to talk about much. (My turn for an overly-cynical guess: since many of them are consultants as you've pointed out, they don't hang around with the same code and the same tests for 7 years, so they don't necessarily see how it really plays out). "Bad tests" are actually a huge net negative for development.
If the test is testing something (like a getter or setter) with basically no chance of breaking, then it's a waste of time. If the test is likely to be fragile or non-deterministic, it's a waste of time. If the test is just too hard to write, and it's not too hard to just test by hand, then automating it is probably a waste of time and you should just QA it by hand every so often.
Finding the right balance tends to come back to the old experience and skill thing: you need to have some intuition about which tests will give you the most value (because that part needs to be rock-solid, or because it's hard to get right, or because it's high-change) and which tests need to be thrown away or never written because they aren't worth it.
Taking any development process too far tends to work out poorly, and taking any metric (like test coverage) too seriously is always a bad idea. That said, I've rarely seen unit testing taken way too far; not testing enough and ending up with buggy, regression-riddled software is a far more common failure mode.
This is why it might be an idea to have unit tests AND QA. Be pragmatic with the unit tests and center them around core functionality and things that are hard to test (think very hard about race conditions for example). QA if they're any good should catch the boneheaded exceptions (such as a mis-behaving getter that calls itself).
Comments
I agree that there's no overarching rule that makes the tradeoffs easy to analyze.
I will, however, say that after working on the same code base for over 7 years now, and watching the company grow from 15 employees to 400, I can't imagine ever working at a place that didn't have a large investment in unit tests, tech debt elimination, automation, etc. Without that stuff, our products almost certainly would have collapsed under their own weight by now, and our ability to ship predictably and on-time would be gone. Even within the company, we have some groups that have done better than others as far as automating tests (both because of team personality and because of technical issues that make certain types of features harder to test), and it's quite obvious that the groups with the best testing are the groups that are able to make much more predictable progress and that are able to ship on time. The groups with less-good testing tend to be prone to fairly massive schedule slippages due to a ton of late-stage regressions that only get caught when they ship their code out to their internal users.
Once things get to the point where no one person can reasonably understand the full implications of their changes, because the system is just too big and complicated, if you don't have unit tests you're in big, big trouble, and you need to reduce tech debt so you can keep things as comprehensible as possible. Even then, of course, you have to constantly decide how much to invest in testing and infrastructure and cleanup versus how much you invest in forward progress, and there's never an obvious equation that will give you a right answer.
If your code base and team are small and likely to stay that way, such that you can still mercilessly refactor and change the code without introducing a bunch of hidden bugs, then testing doesn't matter as much. If you ever expect the code to get to the point where that becomes less true, and where the possibility of introducing errors increases, then it starts to matter a whole lot more.
Hypothetically, let's assume we built the same product with two teams, one that did a bunch of unit testing and one that didn't (call them Team A and Team B). From my experience, what essentially happens is that Team A ships version 1 first, ships version 2 first but takes about as long to build version 2 as Team B, ships version 3 about the same time (since it takes them longer to build it), experiences a massive schedule slip in version 4 (since the complexity catches up to them and things becomes buggy and they start playing whack-a-mole with bugs), and don't really ever ship a version 5 because their code has so much tech debt that no one can change anything safely without breaking something else unintentionally, and they start contemplating a complete rewrite of the code base. Again, totally contrived situation (it doesn't have to go that way, Team B could still totally screw things up anyway, etc.), but that's roughly what I've seen happen, both at my company and at others.
I don't think it's fair to say that consultants push TDD and tech debt reduction because that means the project will take longer: that's a bit overly cynical. Many, many organizations use unit testing and such in house because it has a huge long-term benefit (as well as generally more predictability in the short term, which is often more valuable than absolute speed), not because some consultant told them to do it.
I am not actually against unit tests, but I have seen it being taken to unhealthy extremes. For example at some companies there are automated tools that check that every method has a unit test. In the end people write unit tests for Java getter and setters and so on. Mind numbing as that task is, people also end up writing bad unit tests just to silence the tool.
A lot of unit tests make sense, but I suspect they also offer plenty of opportunities for idling time away.
No question, it's a fine line . . . you need to be pragmatic and ask "is writing and maintaining this test going to save me more time than it costs?" Over several years, the maintenance of the tests themselves becomes a huge cost, which is something the TDD guys don't seem to talk about much. (My turn for an overly-cynical guess: since many of them are consultants as you've pointed out, they don't hang around with the same code and the same tests for 7 years, so they don't necessarily see how it really plays out). "Bad tests" are actually a huge net negative for development.
If the test is testing something (like a getter or setter) with basically no chance of breaking, then it's a waste of time. If the test is likely to be fragile or non-deterministic, it's a waste of time. If the test is just too hard to write, and it's not too hard to just test by hand, then automating it is probably a waste of time and you should just QA it by hand every so often.
Finding the right balance tends to come back to the old experience and skill thing: you need to have some intuition about which tests will give you the most value (because that part needs to be rock-solid, or because it's hard to get right, or because it's high-change) and which tests need to be thrown away or never written because they aren't worth it.
Taking any development process too far tends to work out poorly, and taking any metric (like test coverage) too seriously is always a bad idea. That said, I've rarely seen unit testing taken way too far; not testing enough and ending up with buggy, regression-riddled software is a far more common failure mode.
This is why it might be an idea to have unit tests AND QA. Be pragmatic with the unit tests and center them around core functionality and things that are hard to test (think very hard about race conditions for example). QA if they're any good should catch the boneheaded exceptions (such as a mis-behaving getter that calls itself).