1) Define moderators as subset of voters that you trust. One of such moderators would be you.

2) Rate every other voter against moderators' votes. If voter upvoted the same as moderator, then increase weight of such voter in the future. If voter upvoted against moderator vote -- decrease future voter's weight.

3) For every comment calculate weighted upvote/downvote total based on voters clicks and weights calculated in #2.