Throughput involves both time to process a request and how many requests you can process per unit time.
If you only take 10ms to process a request and give the data back to the user, but then take 200ms afterwards cleaning up after yourself (garbage collection, background tasks etc). then you can only serve up less than 5 requests per second per worker. If you allocate per request 200mb and then free it afterwards and only have 1GB of memory on the server, you can only have a max of 5 workers, so in this case you can only have ~25 requests per second throughput. Fixing either of those cases means you can have a higher throughput in the end without having to scale it out across more servers, since you can prevent the future requests from waiting by either being able to have more workers, or have workers do less work between requests. This isn't even necessarily GC work, it could be sending off jobs to send emails or other jobs that were related to the request regardless of what they were. All of this still ties up the worker that could be handling the request.
Also, the number of requests you get has nothing to do with the number of requests you can actually process. You could be able to process 100k requests per second, but only get 200/second, or vice versa.
Sorry I still don’t understand that - if you have a delay caused by GC before you can respond the next request then this adds time to the request which is waiting during that delay which increases response time.
It depends on what you call response time: if you see it server side, you might count it only from when a request is accepted, from the client side, since the request is sent.
Because you can scale out, it makes sense to measure response time from the server perspective (from when the request is accepted.) Silly example...
Say you have two CPU: you can respond to (CPU bound) requests at a time. When the parallel GC kicks in, it fully occupies one CPU. Response time does not change, but you are handling half of the previous requests per second.
You could scale out, and have two machines handle two requests in parallel when both are running GC.
Comments
Throughput involves both time to process a request and how many requests you can process per unit time.
If you only take 10ms to process a request and give the data back to the user, but then take 200ms afterwards cleaning up after yourself (garbage collection, background tasks etc). then you can only serve up less than 5 requests per second per worker. If you allocate per request 200mb and then free it afterwards and only have 1GB of memory on the server, you can only have a max of 5 workers, so in this case you can only have ~25 requests per second throughput. Fixing either of those cases means you can have a higher throughput in the end without having to scale it out across more servers, since you can prevent the future requests from waiting by either being able to have more workers, or have workers do less work between requests. This isn't even necessarily GC work, it could be sending off jobs to send emails or other jobs that were related to the request regardless of what they were. All of this still ties up the worker that could be handling the request.
Also, the number of requests you get has nothing to do with the number of requests you can actually process. You could be able to process 100k requests per second, but only get 200/second, or vice versa.
Sorry I still don’t understand that - if you have a delay caused by GC before you can respond the next request then this adds time to the request which is waiting during that delay which increases response time.
It depends on what you call response time: if you see it server side, you might count it only from when a request is accepted, from the client side, since the request is sent.
Because you can scale out, it makes sense to measure response time from the server perspective (from when the request is accepted.) Silly example...
Say you have two CPU: you can respond to (CPU bound) requests at a time. When the parallel GC kicks in, it fully occupies one CPU. Response time does not change, but you are handling half of the previous requests per second.
You could scale out, and have two machines handle two requests in parallel when both are running GC.