The problem is that if you discriminate against a minority, then you don't suffer from the limitation of the labor pool very much (since it's a minority). However, that minority can suffer greatly.

So market forces can't hold this sort of behaviour in check, and this can lead to the suppression of an entire culture, which is where the issue lies.