(I think you'd need to timestamp each vote for an improved decay function.)

Well, somewhat. You'd need a timestamped sufficient statistic, but not necessarily each vote.

Consider ACO. Given +1:

  t0=last_change_timestamp
  s0=score_after_last_change
  s1=1+s0/exp(t1-t0)