Bfloat16 is a 16-bit floating-point number format used primarily in machine learning. It allocates one bit to the sign, eight to the exponent, and seven to the stored fraction. This gives it approximately the same numerical range as 32-bit single precision, but substantially less precision. Developed at Google for neural-network workloads, bfloat16 reduces storage, memory traffic, and arithmetic hardware requirements. It is commonly combined with higher-precision accumulation rather than used for every operation in a computation.
Representation
Bfloat16 uses the following bit layout, listed from most significant to least significant bit:
| Field | Width | Purpose |
|---|---|---|
| Sign | 1 bit | Specifies positive or negative |
| Exponent | 8 bits | Encodes the scale, with a bias of 127 |
| Fraction | 7 bits | Encodes the fractional part of the significand |
For a normal finite number, its value is
where is the sign bit, is the unsigned exponent field, and is the unsigned fraction field. Normal numbers use exponent-field values from 1 through 254.
The leading 1 is implicit, so a normal bfloat16 number has eight bits of significand precision, despite storing only seven fraction bits. This corresponds to approximately 2.4 decimal digits of precision; it does not guarantee that every decimal number with that many significant digits is represented exactly.
The boundary exponent values have special meanings:
- : positive or negative zero.
- : a subnormal number, if supported by the implementation.
- : positive or negative infinity.
- : not a number (NaN).
For subnormal encodings, the implicit leading bit is zero, giving
An encoding’s ability to represent subnormals does not imply that every processor preserves them during arithmetic. Some implementations flush subnormal inputs or results to zero.
Range and precision
Bfloat16’s main design choice is to retain the eight-bit exponent of binary32, also called FP32, while shortening its significand.
| Property | Bfloat16 | IEEE binary16, commonly FP16 | IEEE binary32, commonly FP32 |
|---|---|---|---|
| Total bits | 16 | 16 | 32 |
| Exponent bits | 8 | 5 | 8 |
| Stored fraction bits | 7 | 10 | 23 |
| Normal significand precision | 8 bits | 11 bits | 24 bits |
| Smallest positive normal | |||
| Largest finite value, approximately |
Compared with binary16, bfloat16 can represent much larger and much smaller magnitudes, but distinguishes fewer values within a given interval. It therefore exchanges precision for range, not simply accuracy for storage size.
Important bfloat16 values include:
The gap between 1 and the next larger representable value is
Under rounding to nearest, the usual relative rounding-error bound for normal results away from overflow and underflow is , or about 0.39%.
Spacing grows with magnitude. Between 1 and 2, successive values are separated by ; between 128 and 256, they are separated by 1. Every integer from through is exactly representable, but not every integer beyond that interval is.
Bfloat16 and binary32 thus have the same normal exponent limits, but not identical sets of representable values. Binary32 has a slightly larger maximum finite value and supports substantially smaller subnormal values.
Conversion and arithmetic
Relationship to binary32
Bfloat16’s layout matches the most significant 16 bits of a binary32 encoding: the sign bit, all eight exponent bits, and the first seven fraction bits.
At the representation level, a bfloat16 value can be expanded to binary32 by appending 16 zero fraction bits. All finite bfloat16 values are exactly representable in binary32.
The reverse conversion generally loses information. Simply discarding the lower 16 bits truncates the significand; correctly rounded conversion may instead increment the retained portion according to a rounding rule. Common choices include round to nearest, ties to even, although the supported modes depend on the instruction or software implementation.
Conversion also needs appropriate handling of special values. In particular, NaN conversion must preserve a NaN encoding rather than accidentally producing infinity.
Storage precision versus accumulation precision
The datatype used to store operands need not be the datatype used to accumulate results. In mixed-precision computing, bfloat16 inputs are frequently multiplied and accumulated into FP32 results. This is especially important for matrix multiplication and dot products, which add many products:
Using an FP32 accumulator reduces the rounding error that would result from repeatedly rounding the running sum to bfloat16. However, it cannot recover information already lost when the inputs were converted to bfloat16. Hardware may also differ in its treatment of intermediate products, grouping of additions, subnormals, and exceptional values.
Bfloat16 follows several conventions familiar from IEEE 754, including biased exponents and encodings for infinities and NaNs. It is nevertheless distinct from IEEE binary16, and the name alone does not specify every detail of an implementation’s arithmetic behavior.
Use in machine learning
Training
Deep learning often relies on large matrix operations involving model weights, activations, and gradients. Many such workloads tolerate reduced significand precision in parts of the computation, while still requiring a broad range of magnitudes.
Bfloat16 offers several practical advantages:
- Reduced storage: an array occupies half the raw storage of an equivalent FP32 array.
- Reduced memory traffic: moving two-byte elements instead of four-byte elements can reduce bandwidth requirements.
- Efficient arithmetic hardware: a shorter significand can reduce multiplier complexity and support greater arithmetic throughput.
- Broad exponent range: small gradients and large activations are less likely to exceed its normal exponent range than with binary16.
These properties are useful during backpropagation, but they do not make every operation equally suitable for reduced precision. Training systems often retain higher precision for optimizer state, selected reductions, normalization calculations, or parameter updates.
For example, a small update may disappear if a weight and its updated value round to the same bfloat16 number. Keeping an FP32 master copy of weights is one way some mixed-precision training systems address this problem; the exact arrangement varies by framework and optimizer.
Binary16 training commonly uses loss scaling, which rescales a loss function to help prevent small gradients from underflowing. Because bfloat16 has a much wider exponent range, this particular technique is generally unnecessary for bfloat16 training. Its broad range does not, however, prevent all numerical failures.
Inference
Bfloat16 is also used for neural-network inference, including large language models. Model parameters and intermediate activations stored in bfloat16 require less memory than FP32 equivalents.
Bfloat16 is distinct from integer-based neural-network quantization. It retains a floating-point exponent for each value and does not inherently require a separately selected scale for a tensor or group of values. It also normally requires more storage than 8-bit or 4-bit representations.
The precision used for weights, intermediate computations, accumulators, and outputs can differ within a single model execution.
Development and hardware support
The name bfloat16 derives from Brain floating point, reflecting its origin in Google’s Brain research work. It became prominent through Google’s Tensor Processing Units, particularly their use of bfloat16 multiplication with FP32 accumulation.
Support subsequently expanded to central processing units and graphics processing units. Examples include Intel processors with AVX-512 BF16 or appropriate Advanced Matrix Extensions support, and NVIDIA GPUs with suitable Tensor Core support. Availability and throughput depend on the particular processor generation and instruction set.
Frameworks such as TensorFlow and PyTorch expose bfloat16 datatypes and mixed-precision facilities. Software support alone does not guarantee acceleration: unsupported operations may require conversion, execution in another precision, or emulation.
Limitations
Bfloat16’s short significand makes it unsuitable as a universal replacement for FP32 or FP64. Its main numerical limitations include:
- Coarse resolution: nearby values may round to the same representation.
- Loss of small updates: adding a small quantity to a much larger one may leave the stored result unchanged.
- Cancellation: subtracting nearly equal values can expose errors introduced by earlier rounding.
- Accumulation error: long sums can lose substantial accuracy without higher-precision accumulation.
- Implementation differences: rounding, subnormal handling, and operation ordering can affect results across devices.
Sensitive scientific calculations, ill-conditioned problems, and computations requiring tight error bounds may need substantially more precision. Even in machine learning, numerical suitability depends on the model, operations, training configuration, and hardware.
A twofold reduction in element storage also does not imply a twofold improvement in application speed or total memory use. Actual benefits depend on hardware throughput, memory bandwidth, conversion overhead, and the proportion of data that remains in higher precision.