I don't really get this article. It isn't necessary to optimize for minimal allocations if it doesn't affect response time. They mention the new technique was making too many allocations; why do they care how many allocations are being made, if response time is the same?
Good question! Because this is on a hot code path, we were allocating new values faster than the garbage collector was able to remove them. Over extended periods of time, the server would trend toward running out of memory.
They didn't mention having problems with or even measuring throughput in the article, and the number of allocations a program makes is not the only thing that affects throughput. So again, it is unclear why they would spend time optimizing the number of allocations.
Throughput is going to be affected by anything that can bottleneck your application. Even if response time is the same, if you reduce the time spent cleaning up between responses and requests then you can handle a higher number of requests with the same number of workers. If you reduce the amount of memory being allocated you'd also be able to run more workers on the same hardware, also increasing throughput. And as you're implying, reducing time spent making responses would also allow you to increase throughput too.
I still don’t understand that maths - if you both get n requests per second and the time per request has not changed then the throughout is the same isn’t it?
Time between request translates directly into response time as someone is waiting during that time aren’t they? If nobody is waiting and it’s not added to anyone’s wait time then who cares?
The answer really is power consumption during GC even if nobody is waiting, but you didn’t mention that.
Throughput involves both time to process a request and how many requests you can process per unit time.
If you only take 10ms to process a request and give the data back to the user, but then take 200ms afterwards cleaning up after yourself (garbage collection, background tasks etc). then you can only serve up less than 5 requests per second per worker. If you allocate per request 200mb and then free it afterwards and only have 1GB of memory on the server, you can only have a max of 5 workers, so in this case you can only have ~25 requests per second throughput. Fixing either of those cases means you can have a higher throughput in the end without having to scale it out across more servers, since you can prevent the future requests from waiting by either being able to have more workers, or have workers do less work between requests. This isn't even necessarily GC work, it could be sending off jobs to send emails or other jobs that were related to the request regardless of what they were. All of this still ties up the worker that could be handling the request.
Also, the number of requests you get has nothing to do with the number of requests you can actually process. You could be able to process 100k requests per second, but only get 200/second, or vice versa.
Sorry I still don’t understand that - if you have a delay caused by GC before you can respond the next request then this adds time to the request which is waiting during that delay which increases response time.
It depends on what you call response time: if you see it server side, you might count it only from when a request is accepted, from the client side, since the request is sent.
Because you can scale out, it makes sense to measure response time from the server perspective (from when the request is accepted.) Silly example...
Say you have two CPU: you can respond to (CPU bound) requests at a time. When the parallel GC kicks in, it fully occupies one CPU. Response time does not change, but you are handling half of the previous requests per second.
You could scale out, and have two machines handle two requests in parallel when both are running GC.
Comments
I don't really get this article. It isn't necessary to optimize for minimal allocations if it doesn't affect response time. They mention the new technique was making too many allocations; why do they care how many allocations are being made, if response time is the same?
(author here)
Good question! Because this is on a hot code path, we were allocating new values faster than the garbage collector was able to remove them. Over extended periods of time, the server would trend toward running out of memory.
Interesting! That sounds like a garbage collector bug! It's probably worth opening a ticket.
In one work: throughput. More CPU time spent doing useful work rather than GC scanning. Makes you need less scaling out to handle increasing load.
They didn't mention having problems with or even measuring throughput in the article, and the number of allocations a program makes is not the only thing that affects throughput. So again, it is unclear why they would spend time optimizing the number of allocations.
How can you increase throughout without reducing response time?
Throughput is going to be affected by anything that can bottleneck your application. Even if response time is the same, if you reduce the time spent cleaning up between responses and requests then you can handle a higher number of requests with the same number of workers. If you reduce the amount of memory being allocated you'd also be able to run more workers on the same hardware, also increasing throughput. And as you're implying, reducing time spent making responses would also allow you to increase throughput too.
I still don’t understand that maths - if you both get n requests per second and the time per request has not changed then the throughout is the same isn’t it?
Time between request translates directly into response time as someone is waiting during that time aren’t they? If nobody is waiting and it’s not added to anyone’s wait time then who cares?
The answer really is power consumption during GC even if nobody is waiting, but you didn’t mention that.
Throughput involves both time to process a request and how many requests you can process per unit time.
If you only take 10ms to process a request and give the data back to the user, but then take 200ms afterwards cleaning up after yourself (garbage collection, background tasks etc). then you can only serve up less than 5 requests per second per worker. If you allocate per request 200mb and then free it afterwards and only have 1GB of memory on the server, you can only have a max of 5 workers, so in this case you can only have ~25 requests per second throughput. Fixing either of those cases means you can have a higher throughput in the end without having to scale it out across more servers, since you can prevent the future requests from waiting by either being able to have more workers, or have workers do less work between requests. This isn't even necessarily GC work, it could be sending off jobs to send emails or other jobs that were related to the request regardless of what they were. All of this still ties up the worker that could be handling the request.
Also, the number of requests you get has nothing to do with the number of requests you can actually process. You could be able to process 100k requests per second, but only get 200/second, or vice versa.
Sorry I still don’t understand that - if you have a delay caused by GC before you can respond the next request then this adds time to the request which is waiting during that delay which increases response time.
It depends on what you call response time: if you see it server side, you might count it only from when a request is accepted, from the client side, since the request is sent.
Because you can scale out, it makes sense to measure response time from the server perspective (from when the request is accepted.) Silly example...
Say you have two CPU: you can respond to (CPU bound) requests at a time. When the parallel GC kicks in, it fully occupies one CPU. Response time does not change, but you are handling half of the previous requests per second.
You could scale out, and have two machines handle two requests in parallel when both are running GC.