The software standard is. GPU matrix calculations, are not. The hardware, has tiny shifts that rarely matter, except in high finance and... AI modeling.
As the examples demonstrate, although rounding follows deterministic rules, non-associativity introduces nondeterminism, which is further amplified by the larger rounding error of BF16. This becomes particularly relevant in the parallel computations performed in GPUs during LLM inference.
Comments
using which floating point standard? IEEE754 is totally deterministic
The software standard is. GPU matrix calculations, are not. The hardware, has tiny shifts that rarely matter, except in high finance and... AI modeling.
What kind of nonsense is this? Matrix math is entirely deterministic.
Ya might want to just read the paper...