But I would have though the model relied on the higher accuracy during training. So losing that would screw it up.

Yes, during training, where you need to make tiny adjustments to weights. But as far as I understand it inference can still work well because of the sheer number of weights. Give a black-and-white image a high resolution and you can represent any shade of gray if you zoom out a bit.