>The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth.
To me, writing thorough documentation and solid tests from the beginning serves a very important purpose that is often overlooked: it establishes the right type of culture. I think of it as laying the foundation of a building. If you skimp out on materials, the whole thing will be unstable and prone to collapse. And going back later to fix it is often times a lot more costly than doing it right in the first place.
You would never make the argument that a solid foundation is unnecessary if an earthquake never hits, right?
After doing quality for 15 years after another 5 as an application developer, I'm prepared to say that it's an awfully expensive and ineffective way to try to set tone. All it mostly does it get the rest of the company to assume you've got quality all handled and start chucking stuff over the wall at you. It doesn't inspire them to a higher standard.
If QA groups spent half as much time actually testing or helping stabilize development processes as they did maintaining brittle docs they'd probably be considerably more effective. At the end of the day most verbose test docs should have been one set of canonical product docs and a list of inputs/strategies. Instead we repeat and fragment the same info across a bunch of disconnected test docs and then let them rot under the weight of maintenance.
The problem is everyone thinks they have a "one true way." We know that concept is bullshit in software dev and that approach should be tailored to the problem at hand. We understand how important YAGNI and SPOT is, and why sometimes insisting on a perfect architecture is a bad thing in the real world.
Why people think it's any different for QA is beyond me. The same principles largely apply.
Late reply, but on reread I wanted to clarify: solid tests, yes. At the very least, pass needs to mean pass, though you can tolerate some level of spurious failure in some kinds of tests. But if you can't trust a pass, for the scope the test actually tests, better to not have the test and not give false confidence.
It was the replayability part on old packages I was pushing back on. It's certainly a nice to have, but you're more likely to refer back to old test results than actually run the tests again, so having to architect mechanisms to cache and restore old packages, etc, can be overkill. Just cache the results.
As for docs, it's a matter of level of detail. Most QA groups I've been part of or had visibility in overshoot, and generate a lot of docs that aren't particularly useful for any real process. YAGNI applies to docs too.
Further, the docs themselves aren't modular--they usually end up mixing concerns between product documentation and the testing, and in doing so make themselves extremely brittle to product changes whether or not they should really matter to the tests. UI step/verify docs are the absolute worst about this and the most prevalent kind of test documentation.
I've come to the conclusion that most QA groups would be better off just writing one set of canonical docs for the product, if they don't already exist, and then pointing to that with extra info about what inputs or strategy to use for a given test. It's the same SPOT argument for modularizing automation (or any other kind of code), and has the advantage that others can reuse the product docs.
Re: foundation/earthquake argument, there's something to be said for knowing what you can't live without in the case of a disaster, what you can live without with a little bit of pain, and what you can generate JIT if you need it. The former set is usually pretty small. The middle set is where your educated bets lie. The latter set, you should probably never do up front until you've found yourselves having to scramble a couple of times.
Comments
To me, writing thorough documentation and solid tests from the beginning serves a very important purpose that is often overlooked: it establishes the right type of culture. I think of it as laying the foundation of a building. If you skimp out on materials, the whole thing will be unstable and prone to collapse. And going back later to fix it is often times a lot more costly than doing it right in the first place.
You would never make the argument that a solid foundation is unnecessary if an earthquake never hits, right?
After doing quality for 15 years after another 5 as an application developer, I'm prepared to say that it's an awfully expensive and ineffective way to try to set tone. All it mostly does it get the rest of the company to assume you've got quality all handled and start chucking stuff over the wall at you. It doesn't inspire them to a higher standard.
If QA groups spent half as much time actually testing or helping stabilize development processes as they did maintaining brittle docs they'd probably be considerably more effective. At the end of the day most verbose test docs should have been one set of canonical product docs and a list of inputs/strategies. Instead we repeat and fragment the same info across a bunch of disconnected test docs and then let them rot under the weight of maintenance.
The problem is everyone thinks they have a "one true way." We know that concept is bullshit in software dev and that approach should be tailored to the problem at hand. We understand how important YAGNI and SPOT is, and why sometimes insisting on a perfect architecture is a bad thing in the real world.
Why people think it's any different for QA is beyond me. The same principles largely apply.
Late reply, but on reread I wanted to clarify: solid tests, yes. At the very least, pass needs to mean pass, though you can tolerate some level of spurious failure in some kinds of tests. But if you can't trust a pass, for the scope the test actually tests, better to not have the test and not give false confidence.
It was the replayability part on old packages I was pushing back on. It's certainly a nice to have, but you're more likely to refer back to old test results than actually run the tests again, so having to architect mechanisms to cache and restore old packages, etc, can be overkill. Just cache the results.
As for docs, it's a matter of level of detail. Most QA groups I've been part of or had visibility in overshoot, and generate a lot of docs that aren't particularly useful for any real process. YAGNI applies to docs too.
Further, the docs themselves aren't modular--they usually end up mixing concerns between product documentation and the testing, and in doing so make themselves extremely brittle to product changes whether or not they should really matter to the tests. UI step/verify docs are the absolute worst about this and the most prevalent kind of test documentation.
I've come to the conclusion that most QA groups would be better off just writing one set of canonical docs for the product, if they don't already exist, and then pointing to that with extra info about what inputs or strategy to use for a given test. It's the same SPOT argument for modularizing automation (or any other kind of code), and has the advantage that others can reuse the product docs.
Re: foundation/earthquake argument, there's something to be said for knowing what you can't live without in the case of a disaster, what you can live without with a little bit of pain, and what you can generate JIT if you need it. The former set is usually pretty small. The middle set is where your educated bets lie. The latter set, you should probably never do up front until you've found yourselves having to scramble a couple of times.