Right. There is no such thing as downvoting someone's karma. What you vote upon (unless you abuse the system) are individual comments and submissions. Karma is an overall score, not something you edit at will.