To be honest, since opinions with which I disagree often seem stupid, I see downvoting when there is disagreement as a natural reaction to a certain extent.

Philosophically, downvoting should impact the karma of the downvoter.