Either get rid of the whole Karma business or allow everyone to both upvote and downvote. Anything in between is just a hypocrisy.