This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need.
I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are clear, so the trust in any of it is undermined. So then you get a dev culture saying "QA is unnecessary, we can handle all of it," which is generally true at the beginning but can send your company off a cliff if you don't manage transition.
The sad part is that even once quality processes get around that, the all-or-nothing attitude leads to a pyrrhic victory. Quality teams still end up only doing what can be afforded (because math) but everyone thinks they should be doing more. That leads to the pervasive opinion of ineffective quality teams. And, unfortunately, the scramble of trying to do everything at once (and therefore mastering none of it) often makes that a self-fulfilling prophecy.
And I love that it stresses that QA is about hedging bets. Bugs get out, period. The gatekeeper attitude is what leads to the do all the things attitude in the first place. "At any cost" is generally a bad way to strategize.
As for this chart, I'd have added exploratory/informal to seed B2C--exploratory is biggest bang for buck for finding new bugs, and incubating products are -all- new bugs--but I suspect they're lumping what I'd have recommended under dogfooding.
So yeah, very nice. Of course, the best thing to do in any situation is consider the context of your company, product, and market. Think about how maintainable it really needs to be and what would cause the most damage: losing customer money, leaking their data, embarrassing you in the market, eroding their trust. Are there only a few customers to lose or are you mass market and can afford a round of sufficiently obscure failure?
Those are the things to prioritize when you start picking your battles. But this is such a great set of guidelines for where to start that conversation.
Thanks for the good word (I'm the author). I 100% agree about setting the right culture / the difficulty of managing the transition.
It takes a ton of work to establish the right infrastructure + culture. Specifically for the reason you mentioned: a lot of folks see QA in black and white.
As a QA Engineer, I had to create an account to reply to this.
Where are these people saying "the ideal is all the things all the time, we're the gatekeepers,"?
If anyone knows the limitation of QA, it's the people of QA themselves. Trust me, we know that common and non-sensical expectations for "full coverage" and "test everything" are a scourge.
Maybe I'm just lucky to not have worked with the types you describe, but in my experience a QA will be pushing a message of priority-based tradeoffs rather than an unrealistic all-or-nothing approach to quality.
Regarding the article itself, that it advocates being thoughtful about what you test it makes me think that maybe it was written by someone who actually understands QA.
Comments
This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need.
I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are clear, so the trust in any of it is undermined. So then you get a dev culture saying "QA is unnecessary, we can handle all of it," which is generally true at the beginning but can send your company off a cliff if you don't manage transition.
The sad part is that even once quality processes get around that, the all-or-nothing attitude leads to a pyrrhic victory. Quality teams still end up only doing what can be afforded (because math) but everyone thinks they should be doing more. That leads to the pervasive opinion of ineffective quality teams. And, unfortunately, the scramble of trying to do everything at once (and therefore mastering none of it) often makes that a self-fulfilling prophecy.
And I love that it stresses that QA is about hedging bets. Bugs get out, period. The gatekeeper attitude is what leads to the do all the things attitude in the first place. "At any cost" is generally a bad way to strategize.
As for this chart, I'd have added exploratory/informal to seed B2C--exploratory is biggest bang for buck for finding new bugs, and incubating products are -all- new bugs--but I suspect they're lumping what I'd have recommended under dogfooding.
So yeah, very nice. Of course, the best thing to do in any situation is consider the context of your company, product, and market. Think about how maintainable it really needs to be and what would cause the most damage: losing customer money, leaking their data, embarrassing you in the market, eroding their trust. Are there only a few customers to lose or are you mass market and can afford a round of sufficiently obscure failure?
Those are the things to prioritize when you start picking your battles. But this is such a great set of guidelines for where to start that conversation.
(Edits for typos only)
Thanks for the good word (I'm the author). I 100% agree about setting the right culture / the difficulty of managing the transition.
It takes a ton of work to establish the right infrastructure + culture. Specifically for the reason you mentioned: a lot of folks see QA in black and white.
It's doubly hard when you're growing fast.
As a QA Engineer, I had to create an account to reply to this.
Where are these people saying "the ideal is all the things all the time, we're the gatekeepers,"?
If anyone knows the limitation of QA, it's the people of QA themselves. Trust me, we know that common and non-sensical expectations for "full coverage" and "test everything" are a scourge.
Maybe I'm just lucky to not have worked with the types you describe, but in my experience a QA will be pushing a message of priority-based tradeoffs rather than an unrealistic all-or-nothing approach to quality.
Regarding the article itself, that it advocates being thoughtful about what you test it makes me think that maybe it was written by someone who actually understands QA.