In the early days of computing, as much arithmetic as possible was performed using integers, because of the time it took to perform floating point calculations. With the standardisation of floating point formats and the widespread use of dedicated computation units, most of those variables such as display coordinates have become floating point instead of cheap and cheerful integers.

Representation

Floating point numbers come from a continuous range that has to include extremely large positive and negative values, and many very close to zero. They’re most familiar to us from engineering or scientific notation expressing them in terms of a number from 1.0 to almost 10.0, multiplied by a power of ten, such as 1.68301 x 10e-6, which is just above zero at 0.00000168301.

The most widely used form of floating point number in macOS is the Double, which uses 64 bits to encode a number using similar principles to engineering/scientific notation, only the powers used aren’t decimal but binary, making them more difficult to read and understand. In decimal notation, with the radix 10, 0.00000168301 has the significand of 1.68301 and the exponent of -6, making it 1.68301 x 10e-6. As a computer Double, the radix is 2 (binary), so it has a significand of 1.76476389376 and an exponent of -20, making it 1.76476389376e-20.

Some Doubles are exact expressions of the number they’re trying to represent. An obvious example is 1.0, represented as 1.0e0, but even fairly simple numbers like 71.3927 are confusing, with a representation of 1.1155109375e6 (radix 2). To convert between regular decimal floating point and 32- and 64-bit floating point numbers, and their hex representations, my free Mints has a Floating Point Explorer window.

Whether viewed in memory or on disk in raw hex format, floating-point numbers present two formidable challenges to their reading. First, we normally use floating-point numbers with a radix of 10, while their computer representations normally use a radix of 2. The number you and I know as π is around 3.14159274101257324 x 10^0, but in 64-bit floating-point it becomes 1.5707963705062866 x 2^1, which we simply wouldn’t recognise as π.

Set in hexadecimal, though, it gets even worse, as it’s then 4009 21FB 6000 0000, but in 32-bit is 4049 0FDB, which appears completely unrelated.

The reason for the complexity of hex representations of floating-point numbers is that their component parts don’t divide on byte boundaries. In 64-bit doubles, there’s a single sign bit followed by an 11-bit exponent and a 52-bit significand; in 32-bit singles, one sign bit is followed by 8 bits of exponent not aligned to byte boundaries, and a 23-bit significand.

I defy anyone to be able to just look at any floating-point number in hex format and read it off in radix 10 decimal form, apart from a few special examples like +0, which is at least all zeroes. With its Floating Point Explorer, Mints helps you work easily with radix 10 and 2, and with both 32- and 64-bit hex representations.

Simply enter your radix 10 number in the two boxes in the top row, and press the down cursor key, to see that number in radix 2 and in hex. To work in the opposite direction, enter your hex number in the box at the foot, and press the right cursor key to see it in radix 2 and 10 above. For your convenience, the hex can be entered in groups with whitespace between them, such as 4009 21FB 6000 0000, and if needed (which is unusual in floating-point) it will automatically be padded with leading zeroes.

Unlike mathematical numbers, there’s a finite number of different Doubles, and their distribution is far from even. The same Double representing 71.39270000000000 also represents 71.39270000000001, and all the numbers in between them, all but one of which is only an approximation. Around those numbers, there are roughly 70 trillion different floating point numbers per unit (1.0) step in number. These become more dense around zero, and less dense at the extreme ends of the number line. As Doubles become larger in absolute value (disregarding their sign), so they become less precise in absolute but not relative terms.

Errors

Because they’re only approximations, Doubles suffer several problems that can adversely affect calculating with them. These include rounding and cancellation errors.

Rounding errors occur because Doubles have fixed length, so the last place has to be rounded up or down to give the best approximation to the real number. The standard for floating point (IEEE 754) specifies no less than five different rounding functions, that can result in a Double being rounded up or down. Although the relative errors from rounding should be small, they can accumulate in long series of calculations to the point where they affect overall accuracy.

Cancellation errors can be very large, even when only the result of a single operation. This term refers to potentially highly inaccurate results from subtracting numbers that are very close in value. When almost all the digits of the result are lost, these errors can be catastrophic, and may cause the order of calculations to determine the result.

These can be illustrated by two simple calculations, each of which should return a result of exactly 0.0:
((10000000.001 - 10000000.000) - 0.001) * 1.0e8
and
(10000000.001 - (10000000.000 + 0.001)) * 1.0e8
Yet using Swift Doubles, the first returns the incorrect result of 0.016391277311150754. Mints shows a small selection of similar tests in its Double Arithmetic Tests, accessed through the Double entry in the Data item in its Window menu.

With a whole IEEE standard to themselves, floating point numbers have grown their own subdivision of errors and non-errors. The most commonly encountered of these is the NaN, Not a Number, which used to puzzle those plugging through spreadsheets when a formula attempted a heinous crime such as division by zero. The deep joy of NaNs is their propagation: once a NaN creeps into a calculation, it’s likely to turn the whole thing NaN. Then there are two different signed zeroes, +0 and -0, or if you really want a choice, why not have an unsigned zero too, and then decide whether you want all three to be equal or not.

Other formats

Some systems also support extended precision beyond Doubles. One of the advances brought by the first widely used maths coprocessor, Intel’s 8087, was the availability of 80-bit Extended calculations. Although valuable for some, in general, mixing precisions leads to further strange errors that can prove hard to trace. macOS tries to avoid those, and ARM processors don’t have any Extended features, which have to be implemented in additional libraries for those that need them.

Most recently, to accommodate AI using neural networks, smaller floating point numbers have become popular. bfloat16 numbers use only 16 bits of storage, but cover the same range as 32-bit floating point numbers with reduced precision. These promise huge gains in speed by allowing arithmetic instructions on twice the numbers at once, and are supported in CPUs in Apple’s M2 and later chips, and in GPUs.

You’ll occasionally come across other numeric formats, including fixed point and arbitrary precision. These don’t normally have any direct support in general purpose processors, but are implemented in libraries, making them considerably slower and non-transferable. And then there are arrays of numbers in vectors and matrices, complex numbers, and everything else that mathematicians have devised. There is no end.

Further reading

Jean-Michel Muller et al (2018), Handbook of Floating-Point Arithmetic, 2nd ed, Birkhäuser, ISBN 978 3 319 76525 9.
Peter Kornerup and David W Matula (2010), Finite Precision Number Systems and Arithmetic, Cambridge UP, ISBN 978 0 521 76135 2.
Jean-Michel Muller (2006), Elementary Functions, Algorithms and Implementation, 3rd ed, Birkhäuser, ISBN 978 1 4899 7981 0.