You shouldn't downvote a post because you don't agree with it anyway. You should downvote it because it doesn't add anything to the discussion or is otherwise vapid. It follows that there should never be a reason to downvote and reply to a comment.