Limit the amount of downvotes one can perform per day, or per X hours.

Points from comments should have way more weight than karma points from submissions. It's easy to get massive karma points if you submit something interesting. On the other hand, you could spend a lot of time writing a comment and you wouldn't get than many points from it.