I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.
It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client<->server.
All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.
We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.
The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.
It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.
Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.
That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.
A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.
I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.
There's more stuff I've got planned, but that's it for now.
Comments
Good questions.
I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.
It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client<->server.
All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.
We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.
The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.
It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.
Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.
A lot of these features sound similar to MinimaLT: http://cr.yp.to/tcpip/minimalt-20130522.pdf
Do you have explicit DoS protections?
That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.
A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.
I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.
There's more stuff I've got planned, but that's it for now.
It's interesting how much you can streamline a protocol for a single use case. Did you retain the crypto mostly intact-- including the PFS?