Comment on Surpassing vLLM with a Generated Inference StackparentComments−lukebechtelOP6moUnfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well.We believe our improvements would hold on BF16, but let me check.
Comments
Unfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well.
We believe our improvements would hold on BF16, but let me check.