Ah yes. You've got a good point. I find it confusing on how to allow downvotes then. Is age of the account a fair measure? Then what about people who've been here without registering, or people who actually have good discretion.

Maybe enforce a stealth downvote. This would be a case where the downvote button is visible but you use heuristics and probabilistic models to determine to whether actually accept the guys' vote or not, and if yes, what weight to assign it.