You can make this argument on literally any definition - you have to start with something.

So dictionary.com says the defintion of distance is:

the extent or amount of space between two things, points, lines, etc.

That works for me.

I think it makes sense to start with something physical, in the real world that people - and especially kids, can intuatively understand. Our brains by default understand what distance is. Maybe that's true for numbers as well- and that's ok too.