Or just move the freaking down vote button. Change the row layout from

   up/down user time

to

   up user time down

That will make it a lot harder to hit the wrong button.