Indeed, that's fun at the beginning but past one point when you'd want to stop the servers taking 3 connections a day but you can't because these are the 3 most business critical connections, it starts to become much less fun.
Connection establishment in the kernel was an issue before kernel 3.8 I think though not sure. The problem is finding a spare port when you use a lot of concurrent connections and at some point it costs a lot. The "source" directive in haproxy was made to overcome this (you can define a per-server range so that the kernel doesn't have to scan its lists). But nowadays more efficient mechanisms were implemented in the kernel, that we support in haproxy (using IP_BIND_ADDRESS_NO_PORT) and we haven't witnessed these painful moments for a while now.
I was running on FreeBSD, and using port selection inside HAProxy to avoid the kernel selecting it, but still had contention on locks when inserting connections into the table. I can't remember if that was FreeBSD 11 or 12, but the next version made improvements which looked nice, but I wasn't able to test.
Somewhere my (not very great, but working) patch set is floating around, where I added rss aware port selection to HAProxy, so client and origin connections would both hash to the same NIC RX queue, and HAProxy never had to deal with cross-cpu communication for reading sockets.
This interests me. I've long been wondering how we could do that without knowing the NIC's hash and/or without having to brute-force the ports using the NIC's hash to reverse-map them. But maybe there are kernel facilities I'm not aware of to do that. If you happen to stumble upon it and share it to the mailing list, that would be great!
I haven't looked at this code in a long time! Haven't gotten to a computer and it's hard to read this on a phone, so... And I left where I wrote it, so I can't confirm if there were any substantial changes after these. My fuzzy memory is this may be all the HAProxy changes, then I focused on kernel level changes.
Comments
Indeed, that's fun at the beginning but past one point when you'd want to stop the servers taking 3 connections a day but you can't because these are the 3 most business critical connections, it starts to become much less fun.
Connection establishment in the kernel was an issue before kernel 3.8 I think though not sure. The problem is finding a spare port when you use a lot of concurrent connections and at some point it costs a lot. The "source" directive in haproxy was made to overcome this (you can define a per-server range so that the kernel doesn't have to scan its lists). But nowadays more efficient mechanisms were implemented in the kernel, that we support in haproxy (using IP_BIND_ADDRESS_NO_PORT) and we haven't witnessed these painful moments for a while now.
I was running on FreeBSD, and using port selection inside HAProxy to avoid the kernel selecting it, but still had contention on locks when inserting connections into the table. I can't remember if that was FreeBSD 11 or 12, but the next version made improvements which looked nice, but I wasn't able to test.
Somewhere my (not very great, but working) patch set is floating around, where I added rss aware port selection to HAProxy, so client and origin connections would both hash to the same NIC RX queue, and HAProxy never had to deal with cross-cpu communication for reading sockets.
This interests me. I've long been wondering how we could do that without knowing the NIC's hash and/or without having to brute-force the ports using the NIC's hash to reverse-map them. But maybe there are kernel facilities I'm not aware of to do that. If you happen to stumble upon it and share it to the mailing list, that would be great!
https://www.mail-archive.com/haproxy@formilux.org/msg34548.h...
I haven't looked at this code in a long time! Haven't gotten to a computer and it's hard to read this on a phone, so... And I left where I wrote it, so I can't confirm if there were any substantial changes after these. My fuzzy memory is this may be all the HAProxy changes, then I focused on kernel level changes.
Many thanks! Too bad this thread was not noticed and left unresponded. I'll have a look. Thanks again!