I don't think it should, no. There's a big halo effect to karma, and disabling upvotes for stale comments might mitigate it.

But it's not a big problem.