5Gbps could be a single box, if you're ok with all that encompasses.
Otherwise, if you're serving all the traffic with a single IP, you need to do some sort of load balancing. Haproxy + CARP + cold failover is operationally simple, but you lose sessions when your Haproxy box needs maintenance.
ECMP works if your hosting allows for it. You'll still lose sessions during changes though.
Maybe something something pfsync. Or something with proxygen or some other load balancing that came out of Facebook. (I worked there, but not on their load balancers)
More details on what you're planning to do with 5Gbps helps you get advice. I'm assuming https, because cloudflare. Is it mostly static content, or mostly dynamic or mostly proxying? Does it burn a lot of cpu (so you need many boxes anyway) or ? Are you likely to attract DDoS, so you need more inbound bandwidth to accept and drop abuse? Do you expect to provide users with a SLA / what service level do you want to provide? Etc
It's currently operating at 5 Gbps, but it is expected to scale significantly based on traffic trends. It appears to be more cost-effective to rent multiple 5 Gbps boxes rather than a single 20 Gbps box.
The protocol used is protobuf over HTTPS (not gRPC) without channels, and it is stateless. The application proxies requests to underlying services, and is capable of handling 10 Gbps of requests on 8 cores. While the likelihood of a DDoS attack seems low, it is not entirely immune. Though there is no strict SLA, the service aims for availability 99.5% of the time
Your 99.5% target simplifies failover. As long as you don't do it very often, you can just drop sessions; it's not ideal, but it's within your target and just carefully schedule the maitenance (and take the failures when they happen).
If you can, the least equipment option would be to run ECMP from your application servers; no need for a load balancer at all. If you can run 10Gbps on 8 cores, a 16-core server (which is a small server) might do all you need; if so, you could do CARP/VVRP/IP takeover and have everything run through one box at a time.
Or, HAProxy will work fine in front. If you want to make it easier for HAProxy, don't terminate TLS on those, run in tcp mode and let the application server do TLS; but then you can't let HAProxy split requests. If that works for you, you could do better with something that can do Direct Server Return so your load balancer doesn't need to process outbound traffic, but I don't have experience running that kind of load balancer (only being behind one), so no software recommendations.
For HAProxy, especially tcp mode HAProxy, you want a power of two CPU cores that's no more than the number of NIC queues, hyper threads disabled. If you do it right, all the traffic stays on the same CPU, although it's tricky to align the backend connections.
Comments
5Gbps could be a single box, if you're ok with all that encompasses.
Otherwise, if you're serving all the traffic with a single IP, you need to do some sort of load balancing. Haproxy + CARP + cold failover is operationally simple, but you lose sessions when your Haproxy box needs maintenance.
ECMP works if your hosting allows for it. You'll still lose sessions during changes though.
Maybe something something pfsync. Or something with proxygen or some other load balancing that came out of Facebook. (I worked there, but not on their load balancers)
More details on what you're planning to do with 5Gbps helps you get advice. I'm assuming https, because cloudflare. Is it mostly static content, or mostly dynamic or mostly proxying? Does it burn a lot of cpu (so you need many boxes anyway) or ? Are you likely to attract DDoS, so you need more inbound bandwidth to accept and drop abuse? Do you expect to provide users with a SLA / what service level do you want to provide? Etc
Thank you for the answer!
It's currently operating at 5 Gbps, but it is expected to scale significantly based on traffic trends. It appears to be more cost-effective to rent multiple 5 Gbps boxes rather than a single 20 Gbps box. The protocol used is protobuf over HTTPS (not gRPC) without channels, and it is stateless. The application proxies requests to underlying services, and is capable of handling 10 Gbps of requests on 8 cores. While the likelihood of a DDoS attack seems low, it is not entirely immune. Though there is no strict SLA, the service aims for availability 99.5% of the time
Your 99.5% target simplifies failover. As long as you don't do it very often, you can just drop sessions; it's not ideal, but it's within your target and just carefully schedule the maitenance (and take the failures when they happen).
If you can, the least equipment option would be to run ECMP from your application servers; no need for a load balancer at all. If you can run 10Gbps on 8 cores, a 16-core server (which is a small server) might do all you need; if so, you could do CARP/VVRP/IP takeover and have everything run through one box at a time.
Or, HAProxy will work fine in front. If you want to make it easier for HAProxy, don't terminate TLS on those, run in tcp mode and let the application server do TLS; but then you can't let HAProxy split requests. If that works for you, you could do better with something that can do Direct Server Return so your load balancer doesn't need to process outbound traffic, but I don't have experience running that kind of load balancer (only being behind one), so no software recommendations.
For HAProxy, especially tcp mode HAProxy, you want a power of two CPU cores that's no more than the number of NIC queues, hyper threads disabled. If you do it right, all the traffic stays on the same CPU, although it's tricky to align the backend connections.