When I teach about floating point, the two things I try to impress on the students are: it remains a truly incredible engineering feat to believably fit the entire real number line (plus infinities) into 32 or 64 bits, and, it was an incredible political feat to get so many competing companies to agree on one particular way of doing this; both are thanks to Kahan's leadership. Complaints about the quirks of using floating point could be tempered with some appreciation of the hard design decisions that were made, and with gratitude for the people who pulled it off.
Yes. And one take-away for me: when yet another article comes out along the lines of "floating point sucks, and here's a much simpler and better replacement", and the author doesn't mention Kahan and shows in detail that they understand the design tradeoffs and decisions made back then (in IEEE 754), then there's a very good chance that you can toss it.
+1. There are a lot of messy things in the world, and someone comes along as says, "The old way is too messy and complicated, here's a new, simpler, better way," and they don't understand why the old messy thing was messy in the first place. And they don't understand what it was about the old thing that allowed it to last for all those decades.
Somebody posted this anecdote a few days/weeks back, and it stuck with me:
There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”
A lot of the design tradeoffs are not really relevant anymore[0]. There are some ways in which 754 effectively makes a "this is UB, up to the manufacturer" choice (to appease manufacturers of the day) which these days would probably not fly; it's a much easier sell to declare "no ub" (or the equivalent for hw) because we have retrospective power over all the times those were problems, and the hw manufacturers have far LESS power than the application consumer these days.
[0] for example iirc cray had a wonky multiplier, don't remember if it was 754, that (I guess) they thought made it faster but resulted in noncommutative multiplication for many cases.
Some (a lot?) of those got cleaned up in the 2008 revision [1]. And for many practical purposes, once x87 went away as part of x86-64, we now have a world of much more sensible agreement.
We're likely coming back into a period of divergence with ARM vs x86 parts on edge case handling that aren't strictly stated as MUST in the standard (various things related to qNaNs, sNaNs, denormal handling, +-0) but they're minor compared to the "old days".
My guess is they were chunking the multiplier cascade and handling using parallel circuits so that the waveform could be independent across the chunks and not be dependent?
Edit: found it.
2240004 3-24 C (you will want to search by this key, it's a long document!)
"Note that reversing the multiplier and multiplicand operands could cause slightly different results, that is, A x B is not necessarily the same as [B] x A"
I found it instructive to do 8-bit floating point. Keep it simple, 1 bit for sign, 3 bits for exponent, 4 for mantissa, and don't worry about the 'free' extra bit you get on the mantissa. Just compute - exactly! - the values those 256 patterns represent. Now, what pattern of the values emerges? How do the numbers around zero work, vs numbers on the extremes? Are there applications to use your 8-bit floats instead of, say, fixed-point 8-bit representations? Bonus: why do some graphics processors have 16-bit floats?
8-bit floating point numbers (in two variants, mu-law and A-law, which differed in the treatment of denormals, A-law was like IEEE floats, mu-law requires a longer explication), about the same as in your choice, have been used for the digitally-encoded telephone audio signals (PCM audio at 8 kHz sampling rate).
They are probably still used in various older communication equipments.
Using floating-point had the purpose to reduce the quantization noise in comparison to 8-bit fixed-point numbers to the level corresponding to 12-bit or 13-bit fixed-point numbers.
So this was effectively a method of compressing by 50% the bit rate of voice signals.
Later, much better audio compression algorithms have been developed, allowing e.g. a 10 times compression, and such algorithms are used in the modern mobile phones.
Nevertheless, 8-bit floating-point was used for many decades in telephony.
The smallest float you can do that still shows all the patterns is 5-bit (2-bit exponent). If you had only 1 bit for the exponent, you'd go straight from denormal to infinity.
True, it's a combo technical and political achievement -- an extremely rare feat.
Some complaints should be tempered, some others should be flattened into a bare acknowledgement that floating point is simply the wrong tool for the job, e.g. currency.
Maybe one "complaint" that remains is that floating point is too good and displaces progress in development and support for other number formats that are needed in their neiche like bfloat or fixed point.
Some complaints should be tempered, some others should be flattened into a bare acknowledgement that floating point is simply the wrong tool for the job, e.g. currency.
IBM implemented IEEE 854 Radix-10 floating point (which later got subsumed back into IEEE 754) back in the System z9 days.
In fact, testing currency handling is the only thing that Bitcoin is actually useful for, LOL. If you take Bitcoin, you get smacked with a lot of fractional digits that break your currency system right away if you didn't do it right.
That's news to me, apparently 754 was updated in 2008 to include base-10 representations. I didn't even know IEEE standards have revisions at all, multiple mind blows here.
I'm not a mathematician but perhaps one could confirm, this is wrong, no? Irrationals e.g. Pi, sqrt2, cannot be represented. The floating point numbers are like a finite number of teeth on a comb pressed against the continuous real line.
Exactly. Floating-point numbers cannot represent such numbers without error, yet we still see people asserting that floats represent "the entire real number line." The first format to represent such numbers honestly was the original unum format, where the last bit of the fraction indicates if the number is exact or represents the open interval between exact numbers. Like saying pi is 3.14... means pi is between 3.14 and 3.15, a mathematically honest statement. The presence or absence of "..." as a bit in the number was the main idea behind unum arithmetic.
Sure. Obviously, it is only a finite sampling of the real number line, and it is only sampling at a particular set of rational numbers. But it is a sampling with a density that is (roughly) scale invariant. That enables FP computations to have (roughly) the same precision regardless of the overall scaling (choice of units), and that underlies the illusion that you have the whole number line at your disposal when you do computations with FP numbers.
I don't get the purpose of this comment. Yes, it's great that we have IEEE floats standardised across architectures. And yes, it's also true that IEEE floats have serious flaws and that in the intervening decades vastly superior alternatives have been proposed. How are these two assertions incompatible? Are we supposed to be forever thankful for the political and standardisation feat that we never again attempt to improve on a flawed design?
I wonder which are the serious flaws of the IEEE floats in your opinion.
I am familiar with most of the floating-point number formats that have ever been used in computers and there is no doubt that as a general-purpose numeric format all the previous floating-point alternatives have been vastly inferior to the IEEE formats.
For special-purpose niches, usually when a low precision is good enough, it is indeed possible to use some other numeric formats suitable for approximate numbers, e.g. logarithmic numbers, fixed-point numbers, low-precision FP numbers, unums/posits and a few others, which may have certain advantages over the IEEE FP numbers. e.g. a lower cost or higher speed for a given (low) precision, but even for those niches using IEEE FP numbers is usually a decent alternative, not a vastly inferior one.
You mention posits, which are in almost every conceivable aspect superior to IEEE floats. If a similar standardising effort was pushed today we could have vastly better FP in a decade.
For low precision applications posits may be a good choice, sometimes the best.
However it remains to be demonstrated that for high precisions they can be implemented in hardware with similar cost and speed as traditional floating-point numbers.
Regarding the actual advantages of posits, the papers of Gustafson et al. do not inspire much confidence, because besides some correct arguments about genuine advantages of posits, the papers are also full of BS claims, e.g. the claim that posits have the advantage of not generating NaNs.
They do not generate NaNs, because they generate an exception, which is exactly what the IEEE floats also do when you enable trap-on-undefined-operation.
There are also other very dubious claims. While it is very likely that for low precisions posits should be superior and that they might have been a much better alternative to the proliferation of various 16-bit FP formats for machine learning, I have yet to see a single example with posits behaving better than traditional FP formats at high precisions.
For low precisions, there is also the alternative of logarithmic numbers, which allow very fast arithmetic operations but which require look-up tables for addition and subtraction. It is not clear in which applications posits are preferable and in which applications logarithmic numbers are preferable.
This is not true. You base your whole comment on the notion that posits are only good for low precision? This is not correct, for instance 32-bit posits outperform 64-bit floats.
e.g. the claim that posits have the advantage of not generating Nans. They do not generate NaNs, because they generate an exception
This is incorrect also. There are no exceptions in the posit proposal! The point is that e.g. a division by zero is simply a bug in the code, like violating any other ordinary invariant/precondition in ordinary code. If you want exceptions, turn on debugging in your compiler like you would turn on bounds checks on array access for instance. It's silly to require always-on debug assertions and maximum performance at the same time, which is what IEEE floats attempt and fail at.
Regarding NaN, the advantage in posits is that there is only one such value (unsigned infinity), rather than the quadrillions of NaN bit patterns in IEEE floats.
I have yet to see a single example with posits behaving better than traditional FP formats at high precisions.
Again, then you cannot have read the posit proposal thoroughly x) they show that 32-bit floats outperform 64-bit floats in precision, sacrificing only some dynamic range which is useless anyway.
Consider now that most HPC applications nowadays are I/O bound... switching to 32-bit posits increases precision and slashes memory requirements by half. It's immense.
Claiming that a 32-bit numeric format can outperform a 64-bit format is an example of BS claim.
Al 32-bit formats have the same number of points that are distributed over the real numbers, partitioning them in intervals.
The difference in the possible numeric formats is only in the position of the points, so the number of intervals is identical. When for a numeric format the intervals in a certain area are smaller, i.e. the precision is better, that means that in another area the intervals must be larger, so the precision must be worse. Which areas are more important depends on the problem that must be solved.
The posits are just floating-point numbers where the partitioning between the logarithmic part (the exponent) and the linear part (the fraction) is not fixed, but variable.
The posits close to 1 have more fraction bits, while the posits closer to 0 and to infinities have more exponent bits and less fraction bits.
For certain problems 32-bit posits can outperform 32-bit IEEE floats, but 32-bit posits cannot outperform any 64-bit numeric format, because the 64-bit format has billions times more intervals so a 32-bit format does not have any chance of approximating better a number.
Posits can outperform standard floats only at low precision, because in the formats with few bits there are not enough bits to reserve for the exponent, so the exponent range is small, which makes overflows and underflows very likely.
Posits can have a much larger exponent range, while maintaining a good precision close to 1, paying their extended exponent range with reduced precision towards 0 and infinities, i.e. posits have a gradual underflow and overflow, which are more gradual than the IEEE gradual underflow.
At double precision or higher precisions, the IEEE floats have enough exponent range that underflows and overflows become very unlikely, so posits no longer have any advantage.
The standard floating-point numbers have an almost constant relative approximation error over their exponent range.
For most serious numerical work this is the approximation property that is desired. Posits have a variable relative approximation error that becomes worse and worse for small and large numbers. This is normally undesirable. Complex physical models always have both very small numbers and very large numbers that must be approximated well, even better than the numbers close to 1 that are preferred by posits.
Nevertheless for small floating-point numbers, e.g. 16-bit FP, avoiding overflows and underflows becomes more important than the loss of precision at exponent range extremities, so posits are better.
32-bit floats are around the threshold where posits transition from being better to being worse that standard FP numbers.
Depending on the problem, 32-bit floats or 32-bit posits may be better. On the other hand, I have never seen any problem where 64-bit posits, with their worse relative errors, could be better than IEEE double precision.
The original Stanford talk on posits suggested that they generate an exception and not the equivalent of a NaN. A few months later, I changed my mind and the (unique) NaR bit pattern serves the same purpose as NaN does in floats. We have also learned that the best exponent size (es or eS) is 2 bits, independent of the precision of the posit. So there have been some tweaks, but the basic concept is unchanged.
Is an interesting topic and seems to have an interesting history, is there any good books on the topic? Something less academic and dry than I’d expect a floating point book to be but more all encompassing and accessible?
"The End of Error: Unum Computing" is written for a popular audience, not fellow mathematicians. Only high school math is needed, and it's got plenty of humor and full-color illustrations and figures, in an attempt to make a very dry topic into something interesting.
Comments
When I teach about floating point, the two things I try to impress on the students are: it remains a truly incredible engineering feat to believably fit the entire real number line (plus infinities) into 32 or 64 bits, and, it was an incredible political feat to get so many competing companies to agree on one particular way of doing this; both are thanks to Kahan's leadership. Complaints about the quirks of using floating point could be tempered with some appreciation of the hard design decisions that were made, and with gratitude for the people who pulled it off.
Yes. And one take-away for me: when yet another article comes out along the lines of "floating point sucks, and here's a much simpler and better replacement", and the author doesn't mention Kahan and shows in detail that they understand the design tradeoffs and decisions made back then (in IEEE 754), then there's a very good chance that you can toss it.
+1. There are a lot of messy things in the world, and someone comes along as says, "The old way is too messy and complicated, here's a new, simpler, better way," and they don't understand why the old messy thing was messy in the first place. And they don't understand what it was about the old thing that allowed it to last for all those decades.
Somebody posted this anecdote a few days/weeks back, and it stuck with me:
https://fs.blog/chestertons-fence/
A lot of the design tradeoffs are not really relevant anymore[0]. There are some ways in which 754 effectively makes a "this is UB, up to the manufacturer" choice (to appease manufacturers of the day) which these days would probably not fly; it's a much easier sell to declare "no ub" (or the equivalent for hw) because we have retrospective power over all the times those were problems, and the hw manufacturers have far LESS power than the application consumer these days.
[0] for example iirc cray had a wonky multiplier, don't remember if it was 754, that (I guess) they thought made it faster but resulted in noncommutative multiplication for many cases.
Some (a lot?) of those got cleaned up in the 2008 revision [1]. And for many practical purposes, once x87 went away as part of x86-64, we now have a world of much more sensible agreement.
We're likely coming back into a period of divergence with ARM vs x86 parts on edge case handling that aren't strictly stated as MUST in the standard (various things related to qNaNs, sNaNs, denormal handling, +-0) but they're minor compared to the "old days".
[1] https://en.m.wikipedia.org/wiki/IEEE_754-2008_revision
Non-mobile link: https://en.wikipedia.org/wiki/IEEE_754-2008_revision
noncommutative multiplication? what the hell were they doing?
My guess is they were chunking the multiplier cascade and handling using parallel circuits so that the waveform could be independent across the chunks and not be dependent?
Edit: found it.
2240004 3-24 C (you will want to search by this key, it's a long document!)
http://ed-thelen.org/comp-hist/CRAY-1-HardRefMan/CRAY-1-HRM....
"Note that reversing the multiplier and multiplicand operands could cause slightly different results, that is, A x B is not necessarily the same as [B] x A"
Looks like I have some weekend reading to do. Thanks!
I found it instructive to do 8-bit floating point. Keep it simple, 1 bit for sign, 3 bits for exponent, 4 for mantissa, and don't worry about the 'free' extra bit you get on the mantissa. Just compute - exactly! - the values those 256 patterns represent. Now, what pattern of the values emerges? How do the numbers around zero work, vs numbers on the extremes? Are there applications to use your 8-bit floats instead of, say, fixed-point 8-bit representations? Bonus: why do some graphics processors have 16-bit floats?
8-bit floating point numbers (in two variants, mu-law and A-law, which differed in the treatment of denormals, A-law was like IEEE floats, mu-law requires a longer explication), about the same as in your choice, have been used for the digitally-encoded telephone audio signals (PCM audio at 8 kHz sampling rate).
They are probably still used in various older communication equipments.
Using floating-point had the purpose to reduce the quantization noise in comparison to 8-bit fixed-point numbers to the level corresponding to 12-bit or 13-bit fixed-point numbers.
So this was effectively a method of compressing by 50% the bit rate of voice signals.
Later, much better audio compression algorithms have been developed, allowing e.g. a 10 times compression, and such algorithms are used in the modern mobile phones.
Nevertheless, 8-bit floating-point was used for many decades in telephony.
The smallest float you can do that still shows all the patterns is 5-bit (2-bit exponent). If you had only 1 bit for the exponent, you'd go straight from denormal to infinity.
Modern AI hardware is starting to do 8-bit float.
True, it's a combo technical and political achievement -- an extremely rare feat.
Some complaints should be tempered, some others should be flattened into a bare acknowledgement that floating point is simply the wrong tool for the job, e.g. currency.
Maybe one "complaint" that remains is that floating point is too good and displaces progress in development and support for other number formats that are needed in their neiche like bfloat or fixed point.
IBM implemented IEEE 854 Radix-10 floating point (which later got subsumed back into IEEE 754) back in the System z9 days.
In fact, testing currency handling is the only thing that Bitcoin is actually useful for, LOL. If you take Bitcoin, you get smacked with a lot of fractional digits that break your currency system right away if you didn't do it right.
That's news to me, apparently 754 was updated in 2008 to include base-10 representations. I didn't even know IEEE standards have revisions at all, multiple mind blows here.
https://en.m.wikipedia.org/wiki/IEEE_854-1987
Decimal floats are also available in gcc
https://gcc.gnu.org/onlinedocs/gcc/Decimal-Float.html
good point re currency computations. Support for binary-coded decimal is something that deserves to be improved in modern languages (COBOL had it).
I'm not a mathematician but perhaps one could confirm, this is wrong, no? Irrationals e.g. Pi, sqrt2, cannot be represented. The floating point numbers are like a finite number of teeth on a comb pressed against the continuous real line.
Exactly. Floating-point numbers cannot represent such numbers without error, yet we still see people asserting that floats represent "the entire real number line." The first format to represent such numbers honestly was the original unum format, where the last bit of the fraction indicates if the number is exact or represents the open interval between exact numbers. Like saying pi is 3.14... means pi is between 3.14 and 3.15, a mathematically honest statement. The presence or absence of "..." as a bit in the number was the main idea behind unum arithmetic.
Sure. Obviously, it is only a finite sampling of the real number line, and it is only sampling at a particular set of rational numbers. But it is a sampling with a density that is (roughly) scale invariant. That enables FP computations to have (roughly) the same precision regardless of the overall scaling (choice of units), and that underlies the illusion that you have the whole number line at your disposal when you do computations with FP numbers.
I don't get the purpose of this comment. Yes, it's great that we have IEEE floats standardised across architectures. And yes, it's also true that IEEE floats have serious flaws and that in the intervening decades vastly superior alternatives have been proposed. How are these two assertions incompatible? Are we supposed to be forever thankful for the political and standardisation feat that we never again attempt to improve on a flawed design?
I wonder which are the serious flaws of the IEEE floats in your opinion.
I am familiar with most of the floating-point number formats that have ever been used in computers and there is no doubt that as a general-purpose numeric format all the previous floating-point alternatives have been vastly inferior to the IEEE formats.
For special-purpose niches, usually when a low precision is good enough, it is indeed possible to use some other numeric formats suitable for approximate numbers, e.g. logarithmic numbers, fixed-point numbers, low-precision FP numbers, unums/posits and a few others, which may have certain advantages over the IEEE FP numbers. e.g. a lower cost or higher speed for a given (low) precision, but even for those niches using IEEE FP numbers is usually a decent alternative, not a vastly inferior one.
You mention posits, which are in almost every conceivable aspect superior to IEEE floats. If a similar standardising effort was pushed today we could have vastly better FP in a decade.
Unums and posits are ingenious and interesting.
For low precision applications posits may be a good choice, sometimes the best.
However it remains to be demonstrated that for high precisions they can be implemented in hardware with similar cost and speed as traditional floating-point numbers.
Regarding the actual advantages of posits, the papers of Gustafson et al. do not inspire much confidence, because besides some correct arguments about genuine advantages of posits, the papers are also full of BS claims, e.g. the claim that posits have the advantage of not generating NaNs.
They do not generate NaNs, because they generate an exception, which is exactly what the IEEE floats also do when you enable trap-on-undefined-operation.
There are also other very dubious claims. While it is very likely that for low precisions posits should be superior and that they might have been a much better alternative to the proliferation of various 16-bit FP formats for machine learning, I have yet to see a single example with posits behaving better than traditional FP formats at high precisions.
For low precisions, there is also the alternative of logarithmic numbers, which allow very fast arithmetic operations but which require look-up tables for addition and subtraction. It is not clear in which applications posits are preferable and in which applications logarithmic numbers are preferable.
This is not true. You base your whole comment on the notion that posits are only good for low precision? This is not correct, for instance 32-bit posits outperform 64-bit floats.
This is incorrect also. There are no exceptions in the posit proposal! The point is that e.g. a division by zero is simply a bug in the code, like violating any other ordinary invariant/precondition in ordinary code. If you want exceptions, turn on debugging in your compiler like you would turn on bounds checks on array access for instance. It's silly to require always-on debug assertions and maximum performance at the same time, which is what IEEE floats attempt and fail at.
Regarding NaN, the advantage in posits is that there is only one such value (unsigned infinity), rather than the quadrillions of NaN bit patterns in IEEE floats.
Again, then you cannot have read the posit proposal thoroughly x) they show that 32-bit floats outperform 64-bit floats in precision, sacrificing only some dynamic range which is useless anyway.
Consider now that most HPC applications nowadays are I/O bound... switching to 32-bit posits increases precision and slashes memory requirements by half. It's immense.
Claiming that a 32-bit numeric format can outperform a 64-bit format is an example of BS claim.
Al 32-bit formats have the same number of points that are distributed over the real numbers, partitioning them in intervals.
The difference in the possible numeric formats is only in the position of the points, so the number of intervals is identical. When for a numeric format the intervals in a certain area are smaller, i.e. the precision is better, that means that in another area the intervals must be larger, so the precision must be worse. Which areas are more important depends on the problem that must be solved.
The posits are just floating-point numbers where the partitioning between the logarithmic part (the exponent) and the linear part (the fraction) is not fixed, but variable.
The posits close to 1 have more fraction bits, while the posits closer to 0 and to infinities have more exponent bits and less fraction bits.
For certain problems 32-bit posits can outperform 32-bit IEEE floats, but 32-bit posits cannot outperform any 64-bit numeric format, because the 64-bit format has billions times more intervals so a 32-bit format does not have any chance of approximating better a number.
Posits can outperform standard floats only at low precision, because in the formats with few bits there are not enough bits to reserve for the exponent, so the exponent range is small, which makes overflows and underflows very likely.
Posits can have a much larger exponent range, while maintaining a good precision close to 1, paying their extended exponent range with reduced precision towards 0 and infinities, i.e. posits have a gradual underflow and overflow, which are more gradual than the IEEE gradual underflow.
At double precision or higher precisions, the IEEE floats have enough exponent range that underflows and overflows become very unlikely, so posits no longer have any advantage.
The standard floating-point numbers have an almost constant relative approximation error over their exponent range.
For most serious numerical work this is the approximation property that is desired. Posits have a variable relative approximation error that becomes worse and worse for small and large numbers. This is normally undesirable. Complex physical models always have both very small numbers and very large numbers that must be approximated well, even better than the numbers close to 1 that are preferred by posits.
Nevertheless for small floating-point numbers, e.g. 16-bit FP, avoiding overflows and underflows becomes more important than the loss of precision at exponent range extremities, so posits are better.
32-bit floats are around the threshold where posits transition from being better to being worse that standard FP numbers.
Depending on the problem, 32-bit floats or 32-bit posits may be better. On the other hand, I have never seen any problem where 64-bit posits, with their worse relative errors, could be better than IEEE double precision.
The original Stanford talk on posits suggested that they generate an exception and not the equivalent of a NaN. A few months later, I changed my mind and the (unique) NaR bit pattern serves the same purpose as NaN does in floats. We have also learned that the best exponent size (es or eS) is 2 bits, independent of the precision of the posit. So there have been some tweaks, but the basic concept is unchanged.
Many things are possible in our imagination. I'm grateful that my actual world, today, we have working floating point.
Okay, I still don't get the purpose of your comment.
Imagine rust is being built and you reply
:^)
Is an interesting topic and seems to have an interesting history, is there any good books on the topic? Something less academic and dry than I’d expect a floating point book to be but more all encompassing and accessible?
"The End of Error: Unum Computing" is written for a popular audience, not fellow mathematicians. Only high school math is needed, and it's got plenty of humor and full-color illustrations and figures, in an attempt to make a very dry topic into something interesting.
A book on posit arithmetic is in the works.