Maybe downvotes could be split in two.

1. a downvote button, which is available to everyone, but affects only the ranking on the comment list (no graying out due to too many downvotes)

2. a "flag" button, which is available to people with higher karma and serves for moderating.

...because, it seems perfectly human to want to express disagreement with a downvote. I can totally get it, and would/will probably have a hard time restraining myself ^^