Blog: Fundamentals of floating-point numbers and mixed precision training

Efficient training of neural networks is crucial for several reasons: it enables faster developer cycles, optimizes hardware usage (smaller training budget and a reduced carbon footprint) and accelerates the network's response time which is particularly important during inference. Per default, operations on the GPU are performed in float32 precision but float16 precision offers advantages in terms…