Comment on Surpassing vLLM with a Generated Inference StackparentComments−YetAnotherNick6moFor compute bound region(high batch size) yes, but for low batch size it could improve the throughput.
Comments
For compute bound region(high batch size) yes, but for low batch size it could improve the throughput.