I believe if a comment reaches -4 it doesn't go any lower, but every additional downvote is still subtracted from your Karma.