Predictive power is the final arbiter

But how do you measure that predictive power? Humans do have to build an evaluation set. And that evaluation set will be biased one way or the other, you cannot just pretend bias does not exist and hope for the best.