It's never too late to study physics or mathematics. I would however recommend studying physics since it will force you to study the slices of mathematics that was motivated by necessity. A lot of mathematics is genuinely interesting in and of itself, or results in mysteriously rich structures, but has not necessarily found many applications. A lot of mathematics was in fact motivated and (often sloppily) invented by physicists, before being put on more rigorous footing by mathematicians...

Studying physics without learning math is impossible, but physics results in a curricular selection of math which is highly relevant to machine learning.