Comment on Surpassing vLLM with a Generated Inference StackComments−acuozzo6moLuke: Do you have benchmarks for BF16?−lukebechtelOP6moUnfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well.We believe our improvements would hold on BF16, but let me check.
Comments
Luke: Do you have benchmarks for BF16?
Unfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well.
We believe our improvements would hold on BF16, but let me check.