So long as only 10% or less of your incremental successes are mirages, and so long as the downside of shipping a mirage is only a small incremental harm, then shipping 9 success and 1 harm should still get you an overall ~78% win vs just shipping the 9 successes (assuming the bad result is an equivalently negative result to the good ones).
How much are you willing to spend to reduce the downside risk, and how many “good” experiments are you willing to throw away in the process?
Comments
So long as only 10% or less of your incremental successes are mirages, and so long as the downside of shipping a mirage is only a small incremental harm, then shipping 9 success and 1 harm should still get you an overall ~78% win vs just shipping the 9 successes (assuming the bad result is an equivalently negative result to the good ones).
How much are you willing to spend to reduce the downside risk, and how many “good” experiments are you willing to throw away in the process?