Late reply, but on reread I wanted to clarify: solid tests, yes. At the very least, pass needs to mean pass, though you can tolerate some level of spurious failure in some kinds of tests. But if you can't trust a pass, for the scope the test actually tests, better to not have the test and not give false confidence.
It was the replayability part on old packages I was pushing back on. It's certainly a nice to have, but you're more likely to refer back to old test results than actually run the tests again, so having to architect mechanisms to cache and restore old packages, etc, can be overkill. Just cache the results.
As for docs, it's a matter of level of detail. Most QA groups I've been part of or had visibility in overshoot, and generate a lot of docs that aren't particularly useful for any real process. YAGNI applies to docs too.
Further, the docs themselves aren't modular--they usually end up mixing concerns between product documentation and the testing, and in doing so make themselves extremely brittle to product changes whether or not they should really matter to the tests. UI step/verify docs are the absolute worst about this and the most prevalent kind of test documentation.
I've come to the conclusion that most QA groups would be better off just writing one set of canonical docs for the product, if they don't already exist, and then pointing to that with extra info about what inputs or strategy to use for a given test. It's the same SPOT argument for modularizing automation (or any other kind of code), and has the advantage that others can reuse the product docs.
Re: foundation/earthquake argument, there's something to be said for knowing what you can't live without in the case of a disaster, what you can live without with a little bit of pain, and what you can generate JIT if you need it. The former set is usually pretty small. The middle set is where your educated bets lie. The latter set, you should probably never do up front until you've found yourselves having to scramble a couple of times.
Comments
Late reply, but on reread I wanted to clarify: solid tests, yes. At the very least, pass needs to mean pass, though you can tolerate some level of spurious failure in some kinds of tests. But if you can't trust a pass, for the scope the test actually tests, better to not have the test and not give false confidence.
It was the replayability part on old packages I was pushing back on. It's certainly a nice to have, but you're more likely to refer back to old test results than actually run the tests again, so having to architect mechanisms to cache and restore old packages, etc, can be overkill. Just cache the results.
As for docs, it's a matter of level of detail. Most QA groups I've been part of or had visibility in overshoot, and generate a lot of docs that aren't particularly useful for any real process. YAGNI applies to docs too.
Further, the docs themselves aren't modular--they usually end up mixing concerns between product documentation and the testing, and in doing so make themselves extremely brittle to product changes whether or not they should really matter to the tests. UI step/verify docs are the absolute worst about this and the most prevalent kind of test documentation.
I've come to the conclusion that most QA groups would be better off just writing one set of canonical docs for the product, if they don't already exist, and then pointing to that with extra info about what inputs or strategy to use for a given test. It's the same SPOT argument for modularizing automation (or any other kind of code), and has the advantage that others can reuse the product docs.
Re: foundation/earthquake argument, there's something to be said for knowing what you can't live without in the case of a disaster, what you can live without with a little bit of pain, and what you can generate JIT if you need it. The former set is usually pretty small. The middle set is where your educated bets lie. The latter set, you should probably never do up front until you've found yourselves having to scramble a couple of times.