I can only find a reference to what you said with regard to upstream connections. Are you sure this also happens with downstream clients?

https://www.nginx.com/blog/overcoming-ephemeral-port-exhaust...