This is basically what services like cleanfeed[1] are designed to do: send and receive high fidelity audio with minimal latency.

I'm sure in practice you can't have a jam session more than 50ms or so away from your bandmates before the latency starts becoming very noticeable.

[1] https://cleanfeed.net/