Floating point matrix calculations are non-deterministic.
This is not inherent to floating-point math. That actual (true) claim in the article is that different hardware and different hardware configurations produce different results. But deterministic inference is possible, e.g. llama.cpp on CPU is deterministic by default.
The software standard is. GPU matrix calculations, are not. The hardware, has tiny shifts that rarely matter, except in high finance and... AI modeling.
As the examples demonstrate, although rounding follows deterministic rules, non-associativity introduces nondeterminism, which is further amplified by the larger rounding error of BF16. This becomes particularly relevant in the parallel computations performed in GPUs during LLM inference.
Comments
Floating point matrix calculations are non-deterministic. You need to invent new hardware, that doesn't use floating point math, first. [0]
[0] https://arxiv.org/html/2506.09501
This is not inherent to floating-point math. That actual (true) claim in the article is that different hardware and different hardware configurations produce different results. But deterministic inference is possible, e.g. llama.cpp on CPU is deterministic by default.
Right... That's why I added "matrix" in there. Because we don't have matrix operations in software, but in hardware.
Floating point math is entirely deterministic. For a given set of inputs, the same outputs come out, every time.
using which floating point standard? IEEE754 is totally deterministic
The software standard is. GPU matrix calculations, are not. The hardware, has tiny shifts that rarely matter, except in high finance and... AI modeling.
What kind of nonsense is this? Matrix math is entirely deterministic.
Ya might want to just read the paper...