Any reason for doing audio and video work on the backend rather than adopting the (much easier to scale) SFU model and just forwarding packets? Similarly for the physics, seems like doing that work frontend would scale much better.
Even if you use stream forwarding only, network will be a major O(n) cost factor when using cloud providers, at least for video.
There is also some super interesting middle ground between full video forwarding and selective reencoding by using SVC for cheap video resizing on the backend.
SFUs aren't forwarding everything. Using either discrete simulcast layers or SVC they're forwarding one size out of several (usually 3) to each other participant.
Bandwidth will be a high cost no matter what (and anyone planning to scale up real-time video better be prepared to move off the cloud at some point) but needing compute resources for encode/decode on the backend makes large scale realtime video infeasible for any reasonable cost. The only successful large scale deployments push most compute work to the clients.
An extra decode/encode step also adds latency and the total latency budget before UX is impacted is small.
P2P is fine for n=2 (and most videoconferencing systems already try to flow media peer-to-peer in this case), but it often has a pretty severe impact on UX if you go much beyond that.
Even with say n=5, P2P pushes the required upstream bandwidth beyond what is practical with tight realtime constraints on a surprising proportion of home and office connections, especially when Wi-Fi is involved. SFU cuts the upstream bandwidth requirement to be constant rather than a factor of how many other participants are in the conference.
There's also the fact that a significant proportion of offices, and an increasing proportion of homes, simply cannot establish the needed P2P connectivity due to double NAT, firewall, etc. For them you'll be providing a TURN server, which brings back the bandwidth requirement.
The bandwidth costs, if you move off the clouds, are actually quite manageable compared to the compute costs if you try to do decode/encode on the servers.
“If you move off the clouds” can be very expensive. You now need a
globally distributed fleet of servers at different providers and ops and administrative manpower to make sure bills get paid and servers get patched and audited.
Yes, it can be expensive, but nowhere near as expensive as paying per-GiB rates to a cloud for your traffic (which itself is nowhere near as expensive as paying a cloud's rates for compute if you try to do decode/encode in the cloud).
If we are talking about doing realtime video at large scale, we can assume that there is good ops competency in-house. If not, honestly, what are you doing?
Managing fleets of servers across several of the "automation-friendly-but-not-cloud" providers using modern tools is not difficult for some moderately-expensive-but-not-as-expensive-as-cloud-services ops engineers.
Comments
Any reason for doing audio and video work on the backend rather than adopting the (much easier to scale) SFU model and just forwarding packets? Similarly for the physics, seems like doing that work frontend would scale much better.
Even if you use stream forwarding only, network will be a major O(n) cost factor when using cloud providers, at least for video.
There is also some super interesting middle ground between full video forwarding and selective reencoding by using SVC for cheap video resizing on the backend.
SFUs aren't forwarding everything. Using either discrete simulcast layers or SVC they're forwarding one size out of several (usually 3) to each other participant.
Bandwidth will be a high cost no matter what (and anyone planning to scale up real-time video better be prepared to move off the cloud at some point) but needing compute resources for encode/decode on the backend makes large scale realtime video infeasible for any reasonable cost. The only successful large scale deployments push most compute work to the clients.
An extra decode/encode step also adds latency and the total latency budget before UX is impacted is small.
You can avoid bandwidth costs: use peer to peer networking below some number of participant threshold.
P2P is fine for n=2 (and most videoconferencing systems already try to flow media peer-to-peer in this case), but it often has a pretty severe impact on UX if you go much beyond that.
Even with say n=5, P2P pushes the required upstream bandwidth beyond what is practical with tight realtime constraints on a surprising proportion of home and office connections, especially when Wi-Fi is involved. SFU cuts the upstream bandwidth requirement to be constant rather than a factor of how many other participants are in the conference.
There's also the fact that a significant proportion of offices, and an increasing proportion of homes, simply cannot establish the needed P2P connectivity due to double NAT, firewall, etc. For them you'll be providing a TURN server, which brings back the bandwidth requirement.
The bandwidth costs, if you move off the clouds, are actually quite manageable compared to the compute costs if you try to do decode/encode on the servers.
“If you move off the clouds” can be very expensive. You now need a globally distributed fleet of servers at different providers and ops and administrative manpower to make sure bills get paid and servers get patched and audited.
Yes, it can be expensive, but nowhere near as expensive as paying per-GiB rates to a cloud for your traffic (which itself is nowhere near as expensive as paying a cloud's rates for compute if you try to do decode/encode in the cloud).
If we are talking about doing realtime video at large scale, we can assume that there is good ops competency in-house. If not, honestly, what are you doing?
Managing fleets of servers across several of the "automation-friendly-but-not-cloud" providers using modern tools is not difficult for some moderately-expensive-but-not-as-expensive-as-cloud-services ops engineers.