It comes from a time before websites sucked because they are overloaded with ads ads and tracking.
For non-interactively retrieving a single page of HTML, or some other resource, such as a video, or retrieving 100 pages, or 100 videos, in a single TCP connection, without any ads or tracking, HTTP/3 is overkill. It's more complex and it's slower than HTTP/1.0 and HTTP/1.1.
I have a domestic web server, I did implement its code, and the most important was HTTP1.[01] to be very simple to implement, that to lower the cost of implementing my real-life HTTP1.[01] alternative (we all know Big Tech does not like that...).
The best would be to have something like SMTP: the core is extremely simple and yet real-life-works everywhere, and via announced options/extensions it can _optionnaly_ grow in complexity.
It's a bit like drag racing. If all you care about is the performance of a single transfer that doesn't have to deal with packet loss etc, HTTP/1 will win.
Yesterday I was trying to track weird bug. I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.
Google Chrome shows that 6 requests require 2-3 seconds to complete simultaneously. 3 of those requests are tiny static files served by nginx, 3 of those requests are very simple DB queries. Each request completes in few milliseconds using curl, but few seconds in Google Chrome.
Long story short: I wasn't able to track down true source of this obviously wrong behaviour. But I switched ingress-nginx to disable HTTP 2 and with HTTP 1.1 it worked as expected, instantly serving all requests.
I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.
I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.
My guess is it has more to do with resources you probably allocated to your app (especially cpu limit) than any networking overhead which should be negligible in such a trivial setup if done correctly
No, I don't use cpu limits and memory limits are plentiful. Besides, the only difference between working and non-working setups were literally single switch `use-http2` in the ingress controller and it was repeatable experience.
That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.
Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.
Ingress controller is nginx which does reverse proxying. It handles HTTPS termination and it handles routing to the application which is supposed to serve the request. It's either another nginx instance configured to serve static resources or some kind of HTTP server which implements an actual server logic.
I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.
In real life you cannot achieve that level of failure transparency without specifically building your application for it.
If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.
Because kubernetes is much easier to operate than dozens of scripts and docker composes thrown around servers. We have like 5 sites, each site operated by like 5-10 services thrown around 4 servers without any redundancy, autoscaling and even consistency. It's truly is a mess.
What they’re suggesting is that under packet loss conditions QUIC will outperform TCP due to head of line blocking (particularly when there are multiple assets to fetch). Both TCP and QUIC abstract away packet loss but they have different performance characteristics under those conditions.
HTTP/1 has parallelism limitations due to number of simultaneous connections (both in terms of browser and server). HTTP/2 lets you retrieve multiple resources over the same connection improving parallelism but has head of line problems when it does so. HTTP/3 based on Quic solves parallelism and head of line blocking.
The only way the first part of your sentence is correct means that the second part is wrong. HTTP 1 pipelining suffers from head of line blocking just as badly (arguably worse) and the only workaround was opening multiple connections which HTTP/2 also supports.
Given the alternative, switching to a new protocol, requires much more code changes than fixing bad implements of the old protocol, your argument falls quite flat.
There are very real reasons why increasing the number of connections isn’t a thing and that has to do with the expense of creating and managing a connection server side. In fact switching to HTTP/2 was the largest speed up because you could fetch multiple resources at once. The problem of course is that multiplexing resources over a single TCP connection has head of line problems.
Could this have been solved a different way? Perhaps but QUIC is tackling a number of TCP limitations at once (eg protocol ossification due to middleware).
Server-side there is no difference between a different context per TCP connection for a HTTP1 server and a different context per HTTP2 stream within a HTTP2 server.
If anything the HTTP2 requires more state due to the nesting nature of the contexts.
There 100% is. You’re thinking about user space context. But establishing a new socket in the kernel requires a fair amount of under the covers book keeping and the total number of file descriptors keeping those resources alive is a real problem for very popular web services that have to handle lots of concurrent simultaneous connections. At the application layer it’s less of an issue.
HTTP/2 is indeed more complex to maintain in the application layer but that’s less to do with the memory implications of multiplexing requests /responses over a single TCP connection.
I know you think you're coming off smarter than everyone else, but it's not how it's landing. Turns out things are not that overly reductive to that extent at all in the real world
I'm not sure what you mean by "apologist" or what you're trying to say, I'm not the person you're answering to, and I have no dog in this fight.
But you're talking in a very affirmative manner and apparently trying to say that one is "correct" and the other is not, while absolutely ignoring context and usage.
I recommend you either run yourself, or find on the web, a benchmark about not a single HTTP request, but an actual web page, requesting the html, the css, the js, and the images. Don't even need to go modern web, even any old pre 2010 design with no font or anything else fancy will do.
You will see that HTTP 1 and 1.1 are way, way worse at it that HTTP 2. Which is why HTTP 2 was created, and why it should be used. Also the sad joke that was domain rolling to trick simultaneous request per host configurations.
Overall, your point of view doesn't make sense because this is not a winner takes all game. Plenty of server should and do also run HTTP 1 for their usage, notably file servers and the likes. The question to ask is "how many request in parallel do the user need, and how important that they all finish as close to each other as possible instead of one after the other".
You can transfer as many requests in parallel with HTTP/1.1 as you like by simply establishing more TCP connections to the server. The problem is that browsers traditionally limited the number of concurrent connections per server to 3. There’s also a speed penalty incurred with new connections to a host since initial TCP window sizes start out small, but it’s unclear whether that initial speed penalty significantly degrades the user experience.
The fact that anyone running wrk or hey can coerce a web server to produce hundreds of thousands of RPS and saturate 100Gb links with plain old HTTP/1.1 with connection reuse and parallel threads (assuming of course that your load tester, server, and network are powerful enough) ought to be enough to convince anyone that the protocol is more than capable.
But whether it’s the best one for the real world of thousands of different consumer device agents, flaky networks with huge throughput and latency and error/drop rates, etc. is a different question indeed, and these newer protocols may in fact provide better overall user experiences. Protocols that work well under perfect conditions may not be the right ones for imperfect conditions.
TCP connections are bottlenecked not just by the browser/client, but also at the load-balancer/server. Modulo SO_REUSEPORT, a server can maintain at most 64k active connections, which is far below any reasonable expectation for capacity of concurrent requests. You have to decouple application-level requests from physical-level connections to get any kind of reasonable performance out of a protocol. This has been pretty well understood for decades.
That limitation was overcome over 20 years ago with the invention of Direct Server Return (DSR) technology, since the remote IP becomes that of the actual client. (This also helped relieve pressure on load balancers since they don't need to process return traffic.) Another way to overcome this would be to use netblocks instead of IP addresses on both the load balancers (upstream side) and the server side (so the LB has multiple IPs to connect to and the server to respond from).
The benefit of DSR became mitigated a bit after CGNAT (in the IPv4 space anyway) began to be rolled out, since it can masquerade a large group of clients behind a single IP address. (CGNAT poses other, different problems related to fraud and abuse mitigation.)
Which limit, exactly, are you referring to? Both load balancers and backend servers can juggle millions of concurrent connections nowadays. You mentioned a 64k connection limit but that’s not a hard file descriptor limit, nor does the 65536 port limit apply if the source and/or destination IPs differ.
Both load balancers and backend servers can juggle millions of concurrent connections nowadays.
Maybe with SO_REUSEPORT, but not in general.
A TCP connection is identified by a 5-tuple that requires a unique port for both the client and server. TCP represents ports as uint16s, which means the max number of possible ports per address is 65536.
tl;dr: 1 server IP address = no more than 64k incoming connections
Yes, I'm aware that 4-tuples must be unique. And so, by having a LB and/or server bind to more than one IP address, you can easily overcome that limit.
It's quite common in Kubernetes deployments, where each server in a separate Pod binds to a separate IP address.
And, as I said before, with DSR, there's a broad diversity of client IPs, so a single server address doesn't typically cause concerns with 4-tuple exhaustion.
That's a lot of mays. One might imagine that before this stuff becomes the latest version of an internet standard, these theoretical qualifications might be proven out, to estimate its impact on the world at large. But it was useful to one massive corporation, so I guess that makes it good enough to supplant what came before for the whole web.
Google did a great deal of research on the question using real-world telemetry before trying it in Chrome and proposing it as a standard to the IETF’s working group. And others including Microsoft and Facebook gave feedback; it wasn’t iterated on in a vacuum. The history is open and well documented and there are metrics that support it. See e.g. https://www.chromium.org/spdy/spdy-whitepaper/
HTTP/1 remains the one with the highest bandwidth.
To be fair, HTTP/2 and HTTP/3 weren't exactly aimed at maximizing bandwidth. They were focused on mitigating the performance constraints of having to spawn dozens of connections to perform the dozens of requests required to open a single webpage.
Couldn't agree more. So many performance problems could be mitigated if people wrote their client/server code to make as few requests as possible.
Consider the case of requesting a webpage with hundreds of small images, one should embed all of the images into the single webpage! Requiring each image to be fetched in a different http request is ridiculous. It pains me to look at the network tab of modern websites.
I don't think it's realistic to expect a page load to not make a bunch of requests, considering that you will always have to support use cases involving downloading many small images. Either you handle that by expecting your servers to open a dedicated connection for each download request, or you take steps for that not to be an issue. Even if you presume horizontal scaling could mitigate that problem from the server side, you cannot sidestep the fact that you could simply reuse a single connection to get all your resources, or not require a connection at all.
Honestly, one would think that the switch to a binary protocol and then to a different transport layer protocol would be justified by massive gains in performance...
I interpreted his comment as saying “where’s the 20% speed up” which seems like a more reasonable interpretation in context. A 20% speed up is actually quite substantial because that’s aggregate - it must mean there’s situations where it’s more as well as situations where it’s unchanged or worse (although unlikely to be worse under internet conditions).
Comments
HTTP/1 remains the one with the highest bandwidth.
No surprise here.
It comes from a time before websites sucked because they are overloaded with ads ads and tracking.
For non-interactively retrieving a single page of HTML, or some other resource, such as a video, or retrieving 100 pages, or 100 videos, in a single TCP connection, without any ads or tracking, HTTP/3 is overkill. It's more complex and it's slower than HTTP/1.0 and HTTP/1.1.
I have a domestic web server, I did implement its code, and the most important was HTTP1.[01] to be very simple to implement, that to lower the cost of implementing my real-life HTTP1.[01] alternative (we all know Big Tech does not like that...).
The best would be to have something like SMTP: the core is extremely simple and yet real-life-works everywhere, and via announced options/extensions it can _optionnaly_ grow in complexity.
It's a bit like drag racing. If all you care about is the performance of a single transfer that doesn't have to deal with packet loss etc, HTTP/1 will win.
Yesterday I was trying to track weird bug. I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.
Google Chrome shows that 6 requests require 2-3 seconds to complete simultaneously. 3 of those requests are tiny static files served by nginx, 3 of those requests are very simple DB queries. Each request completes in few milliseconds using curl, but few seconds in Google Chrome.
Long story short: I wasn't able to track down true source of this obviously wrong behaviour. But I switched ingress-nginx to disable HTTP 2 and with HTTP 1.1 it worked as expected, instantly serving all requests.
I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.
My guess is it has more to do with resources you probably allocated to your app (especially cpu limit) than any networking overhead which should be negligible in such a trivial setup if done correctly
No, I don't use cpu limits and memory limits are plentiful. Besides, the only difference between working and non-working setups were literally single switch `use-http2` in the ingress controller and it was repeatable experience.
That is indeed interesting. I’ve used nginx with grpc (which uses http2) years ago and definitely did not see any such latency issues even at 50k qps
Kubernetes makes everything slow and complicated.
Why do you even need to have proxies or load balancers in between? Another invention of the web cloud people.
That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.
Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.
Ingress controller is nginx which does reverse proxying. It handles HTTPS termination and it handles routing to the application which is supposed to serve the request. It's either another nginx instance configured to serve static resources or some kind of HTTP server which implements an actual server logic.
I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.
In real life you cannot achieve that level of failure transparency without specifically building your application for it.
If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.
nginx is notoriously bad at not-HTTP 1.1. I wouldn't even bother trying.
Envoy is significantly better in this department.
Huh TIL. I had always considered nginx the “fast” http server.
Nginx is fast enough for most applications. I wouldn’t bother switching if you dont need the power of “software defined” proxy
Performance aside, if you need fully functional HTTP/2 and maybe HTTP/3 support then you're better off not using nginx. It's a pain.
What were your reasons for moving the site to kubernetes?
Because kubernetes is much easier to operate than dozens of scripts and docker composes thrown around servers. We have like 5 sites, each site operated by like 5-10 services thrown around 4 servers without any redundancy, autoscaling and even consistency. It's truly is a mess.
It runs over TCP, you don't need to deal with packet loss.
What they’re suggesting is that under packet loss conditions QUIC will outperform TCP due to head of line blocking (particularly when there are multiple assets to fetch). Both TCP and QUIC abstract away packet loss but they have different performance characteristics under those conditions.
HTTP/1 doesn't have head of line blocking, only HTTP/2 does.
HTTP/1 has parallelism limitations due to number of simultaneous connections (both in terms of browser and server). HTTP/2 lets you retrieve multiple resources over the same connection improving parallelism but has head of line problems when it does so. HTTP/3 based on Quic solves parallelism and head of line blocking.
The only way the first part of your sentence is correct means that the second part is wrong. HTTP 1 pipelining suffers from head of line blocking just as badly (arguably worse) and the only workaround was opening multiple connections which HTTP/2 also supports.
Most browsers only support very limited number of connections so it kinda does
Limitations of certain implementations are irrelevant.
The protocol does not have any such limitation.
Totally, after all it’s not like we live in a real world where these things matter
Given the alternative, switching to a new protocol, requires much more code changes than fixing bad implements of the old protocol, your argument falls quite flat.
There are very real reasons why increasing the number of connections isn’t a thing and that has to do with the expense of creating and managing a connection server side. In fact switching to HTTP/2 was the largest speed up because you could fetch multiple resources at once. The problem of course is that multiplexing resources over a single TCP connection has head of line problems.
Could this have been solved a different way? Perhaps but QUIC is tackling a number of TCP limitations at once (eg protocol ossification due to middleware).
Server-side there is no difference between a different context per TCP connection for a HTTP1 server and a different context per HTTP2 stream within a HTTP2 server.
If anything the HTTP2 requires more state due to the nesting nature of the contexts.
There 100% is. You’re thinking about user space context. But establishing a new socket in the kernel requires a fair amount of under the covers book keeping and the total number of file descriptors keeping those resources alive is a real problem for very popular web services that have to handle lots of concurrent simultaneous connections. At the application layer it’s less of an issue.
HTTP/2 is indeed more complex to maintain in the application layer but that’s less to do with the memory implications of multiplexing requests /responses over a single TCP connection.
There is no requirement for TCP to be implemented in the kernel; if anything all of the highest-performing implementations are in userland.
It's a mystery, it's almost as if people have spent decades optimizing it.
Or rather, it was simply designed correctly.
I know you think you're coming off smarter than everyone else, but it's not how it's landing. Turns out things are not that overly reductive to that extent at all in the real world
Not everyone else, just Google and other HTTP/2 apologists.
I'm not sure what you mean by "apologist" or what you're trying to say, I'm not the person you're answering to, and I have no dog in this fight.
But you're talking in a very affirmative manner and apparently trying to say that one is "correct" and the other is not, while absolutely ignoring context and usage.
I recommend you either run yourself, or find on the web, a benchmark about not a single HTTP request, but an actual web page, requesting the html, the css, the js, and the images. Don't even need to go modern web, even any old pre 2010 design with no font or anything else fancy will do.
You will see that HTTP 1 and 1.1 are way, way worse at it that HTTP 2. Which is why HTTP 2 was created, and why it should be used. Also the sad joke that was domain rolling to trick simultaneous request per host configurations.
Overall, your point of view doesn't make sense because this is not a winner takes all game. Plenty of server should and do also run HTTP 1 for their usage, notably file servers and the likes. The question to ask is "how many request in parallel do the user need, and how important that they all finish as close to each other as possible instead of one after the other".
Similarily, HTTP3 is mostly about latency.
You can transfer as many requests in parallel with HTTP/1.1 as you like by simply establishing more TCP connections to the server. The problem is that browsers traditionally limited the number of concurrent connections per server to 3. There’s also a speed penalty incurred with new connections to a host since initial TCP window sizes start out small, but it’s unclear whether that initial speed penalty significantly degrades the user experience.
The fact that anyone running wrk or hey can coerce a web server to produce hundreds of thousands of RPS and saturate 100Gb links with plain old HTTP/1.1 with connection reuse and parallel threads (assuming of course that your load tester, server, and network are powerful enough) ought to be enough to convince anyone that the protocol is more than capable.
But whether it’s the best one for the real world of thousands of different consumer device agents, flaky networks with huge throughput and latency and error/drop rates, etc. is a different question indeed, and these newer protocols may in fact provide better overall user experiences. Protocols that work well under perfect conditions may not be the right ones for imperfect conditions.
TCP connections are bottlenecked not just by the browser/client, but also at the load-balancer/server. Modulo SO_REUSEPORT, a server can maintain at most 64k active connections, which is far below any reasonable expectation for capacity of concurrent requests. You have to decouple application-level requests from physical-level connections to get any kind of reasonable performance out of a protocol. This has been pretty well understood for decades.
That limitation was overcome over 20 years ago with the invention of Direct Server Return (DSR) technology, since the remote IP becomes that of the actual client. (This also helped relieve pressure on load balancers since they don't need to process return traffic.) Another way to overcome this would be to use netblocks instead of IP addresses on both the load balancers (upstream side) and the server side (so the LB has multiple IPs to connect to and the server to respond from).
The benefit of DSR became mitigated a bit after CGNAT (in the IPv4 space anyway) began to be rolled out, since it can masquerade a large group of clients behind a single IP address. (CGNAT poses other, different problems related to fraud and abuse mitigation.)
It's not a question of IP addresses, it's about connections.
Which limit, exactly, are you referring to? Both load balancers and backend servers can juggle millions of concurrent connections nowadays. You mentioned a 64k connection limit but that’s not a hard file descriptor limit, nor does the 65536 port limit apply if the source and/or destination IPs differ.
Maybe with SO_REUSEPORT, but not in general.
A TCP connection is identified by a 5-tuple that requires a unique port for both the client and server. TCP represents ports as uint16s, which means the max number of possible ports per address is 65536.
tl;dr: 1 server IP address = no more than 64k incoming connections
Yes, I'm aware that 4-tuples must be unique. And so, by having a LB and/or server bind to more than one IP address, you can easily overcome that limit.
If you have multiple client-accessible IP addresses available to a server, I guess? But that's in no way common?
It's quite common in Kubernetes deployments, where each server in a separate Pod binds to a separate IP address.
And, as I said before, with DSR, there's a broad diversity of client IPs, so a single server address doesn't typically cause concerns with 4-tuple exhaustion.
I think we're speaking from wildly different, and incompatible, contexts.
DSR is niche at best, not something that can be assumed.
That's a lot of mays. One might imagine that before this stuff becomes the latest version of an internet standard, these theoretical qualifications might be proven out, to estimate its impact on the world at large. But it was useful to one massive corporation, so I guess that makes it good enough to supplant what came before for the whole web.
HTTP/2 or /3 were never about optimizing bandwidth, but latency.
Google did a great deal of research on the question using real-world telemetry before trying it in Chrome and proposing it as a standard to the IETF’s working group. And others including Microsoft and Facebook gave feedback; it wasn’t iterated on in a vacuum. The history is open and well documented and there are metrics that support it. See e.g. https://www.chromium.org/spdy/spdy-whitepaper/
To be fair, HTTP/2 and HTTP/3 weren't exactly aimed at maximizing bandwidth. They were focused on mitigating the performance constraints of having to spawn dozens of connections to perform the dozens of requests required to open a single webpage.
Too bad that the alternative option - not requiring dozens of requests just for initial rendering of a single page - didn't catch on.
Couldn't agree more. So many performance problems could be mitigated if people wrote their client/server code to make as few requests as possible.
Consider the case of requesting a webpage with hundreds of small images, one should embed all of the images into the single webpage! Requiring each image to be fetched in a different http request is ridiculous. It pains me to look at the network tab of modern websites.
I don't think it's realistic to expect a page load to not make a bunch of requests, considering that you will always have to support use cases involving downloading many small images. Either you handle that by expecting your servers to open a dedicated connection for each download request, or you take steps for that not to be an issue. Even if you presume horizontal scaling could mitigate that problem from the server side, you cannot sidestep the fact that you could simply reuse a single connection to get all your resources, or not require a connection at all.
HTTP3 also wants to minimize latency in bad network environments, not just mitigating the issue of too many requests.
Assuming SSL, HTTP/1 does not deliver better throughput than HTTP/2 in general.
I'm not sure why you believe otherwise. Got any references?
Honestly, one would think that the switch to a binary protocol and then to a different transport layer protocol would be justified by massive gains in performance...
It's definitely not a given that a binary protocol + etc. will yield a "massive gain in performance" versus naive e.g. gzipped JSON over HTTP.
The website being tested probably isn’t complicated enough to demonstrate that difference.
Even then, I remember the sales pitches all mentioning performance improvements in the order of about 10-20%
In real world conditions, not loop back synthetic benchmarks.
I believe his point is 10-20% gain is not massive.
FWIW I don't know if that is what was claimed.
20% is what real world data from Google suggested: https://www.zdnet.com/article/google-speeds-up-the-web-with-...
I interpreted his comment as saying “where’s the 20% speed up” which seems like a more reasonable interpretation in context. A 20% speed up is actually quite substantial because that’s aggregate - it must mean there’s situations where it’s more as well as situations where it’s unchanged or worse (although unlikely to be worse under internet conditions).