(where there is already network delay an order of magnitude larger than the scheduler would impart)

Not the right way to think about it; the connection latency is irrelevant. What is relevant is that you need to play audio in sync with the video, and that audio is coming to you approximately simultaneously with the video it's meant to be synced with.