Hardware support will vary widely, as will speed on these smaller FP formats, sometimes intentionally nerfed in consumer cards.
Lots of devices with embedded "AI accelerators" will also only do things like INT8, and for some reason INT8 is generally worse than the same size FP8 (maybe that could be fixed with smarter quantization).
Comments
Hardware support will vary widely, as will speed on these smaller FP formats, sometimes intentionally nerfed in consumer cards.
Lots of devices with embedded "AI accelerators" will also only do things like INT8, and for some reason INT8 is generally worse than the same size FP8 (maybe that could be fixed with smarter quantization).