If you look at it, it's quite sensitive enough to follow finger motions, not only whole hand motions. So it could even follow your fingers as your hands rest on a desk.

I agree, you don't need to use your whole arm. It's like playing tennis on the Wii. You don't have to flail your arms around. You just need enough motion for the accelerometers.