If by "connection", you mean TCP, probably not. But that's not the only way to maintain connections, and it's certainly not the only reliable network protocol.
My latest project keeps every "connection" open at all times, but it's a custom UDP based reliable messaging protocol, not TCP. At Facebook's scale, we'd have the equivalent of one billion connections "open". It's easy to keep them open, despite changing IP addresses, because every packet is public-key authenticated and encrypted, so you don't have to rely on IP addresses to know who you're talking to...
It also means you only pay for a connection setup time once. For mobile devices, the improvement in latency is palpable.
Can I ask some questions about that? Extremely interested..
I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.)
I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc.
Are you only using UDP from device->infrastructure? If not, do you rely on the devices providing an ip:port over a heartbeat of sorts (to keep up with IP changes?)
Do you have any issues with deliverability, in either direction? (not due to UDP's inherent properties, but because of carrier network behavior)
thanks very much for anything you're able to answer.
I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.
It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client<->server.
All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.
We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.
The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.
It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.
Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.
That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.
A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.
I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.
There's more stuff I've got planned, but that's it for now.
Comments
If by "connection", you mean TCP, probably not. But that's not the only way to maintain connections, and it's certainly not the only reliable network protocol.
My latest project keeps every "connection" open at all times, but it's a custom UDP based reliable messaging protocol, not TCP. At Facebook's scale, we'd have the equivalent of one billion connections "open". It's easy to keep them open, despite changing IP addresses, because every packet is public-key authenticated and encrypted, so you don't have to rely on IP addresses to know who you're talking to...
It also means you only pay for a connection setup time once. For mobile devices, the improvement in latency is palpable.
Can I ask some questions about that? Extremely interested..
I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.)
I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc.
Are you only using UDP from device->infrastructure? If not, do you rely on the devices providing an ip:port over a heartbeat of sorts (to keep up with IP changes?)
Do you have any issues with deliverability, in either direction? (not due to UDP's inherent properties, but because of carrier network behavior)
thanks very much for anything you're able to answer.
Good questions.
I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.
It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client<->server.
All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.
We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.
The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.
It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.
Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.
A lot of these features sound similar to MinimaLT: http://cr.yp.to/tcpip/minimalt-20130522.pdf
Do you have explicit DoS protections?
That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.
A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.
I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.
There's more stuff I've got planned, but that's it for now.
It's interesting how much you can streamline a protocol for a single use case. Did you retain the crypto mostly intact-- including the PFS?