{
  "id": "bfloat16",
  "title": "Bfloat16",
  "lang": "en",
  "source_lang": "en",
  "summary": "Bfloat16 is a 16-bit floating-point format that preserves the exponent range of binary32 while reducing precision, widely used in machine learning.",
  "category": "computer-science",
  "subcategory": "hardware",
  "type": "concept",
  "aliases": [
    "BF16",
    "Brain floating point",
    "Brain floating-point format"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/bfloat16/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/bfloat16/",
      "json": "https://aiwiki.page/zh/bfloat16.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Bfloat16 is a 16-bit floating-point number format used primarily in machine learning. It allocates one bit to the sign, eight to the exponent, and seven to the stored fraction. This gives it approximately the same numerical range as 32-bit single precision, but substantially less precision. Developed at Google for neural-network workloads, bfloat16 reduces storage, memory traffic, and arithmetic hardware requirements. It is commonly combined with higher-precision accumulation rather than used for every operation in a computation."
    },
    {
      "heading": "Representation",
      "text": "Bfloat16 uses the following bit layout, listed from most significant to least significant bit:\n\nField | Width | Purpose\nSign | 1 bit | Specifies positive or negative\nExponent | 8 bits | Encodes the scale, with a bias of 127\nFraction | 7 bits | Encodes the fractional part of the significand\n\nFor a normal finite number, its value is\n\n\\[x=(-1)^s\\,2^{E-127}\\left(1+\\frac{F}{2^7}\\right),\\]\n\nwhere \\(s\\) is the sign bit, \\(E\\) is the unsigned exponent field, and \\(F\\) is the unsigned fraction field. Normal numbers use exponent-field values from 1 through 254.\n\nThe leading 1 is implicit, so a normal bfloat16 number has eight bits of significand precision, despite storing only seven fraction bits. This corresponds to approximately 2.4 decimal digits of precision; it does not guarantee that every decimal number with that many significant digits is represented exactly.\n\nThe boundary exponent values have special meanings:\n\n- \\(E=0,\\ F=0\\): positive or negative zero.\n- \\(E=0,\\ F\\ne0\\): a subnormal number, if supported by the implementation.\n- \\(E=255,\\ F=0\\): positive or negative infinity.\n- \\(E=255,\\ F\\ne0\\): not a number (NaN).\n\nFor subnormal encodings, the implicit leading bit is zero, giving\n\n\\[x=(-1)^s\\,2^{-126}\\frac{F}{2^7}.\\]\n\nAn encoding’s ability to represent subnormals does not imply that every processor preserves them during arithmetic. Some implementations flush subnormal inputs or results to zero."
    },
    {
      "heading": "Range and precision",
      "text": "Bfloat16’s main design choice is to retain the eight-bit exponent of binary32, also called FP32, while shortening its significand.\n\nProperty | Bfloat16 | IEEE binary16, commonly FP16 | IEEE binary32, commonly FP32\nTotal bits | 16 | 16 | 32\nExponent bits | 8 | 5 | 8\nStored fraction bits | 7 | 10 | 23\nNormal significand precision | 8 bits | 11 bits | 24 bits\nSmallest positive normal | \\(2^{-126}\\) | \\(2^{-14}\\) | \\(2^{-126}\\)\nLargest finite value, approximately | \\(3.39\\times10^{38}\\) | \\(6.55\\times10^4\\) | \\(3.40\\times10^{38}\\)\n\nCompared with binary16, bfloat16 can represent much larger and much smaller magnitudes, but distinguishes fewer values within a given interval. It therefore exchanges precision for range, not simply accuracy for storage size.\n\nImportant bfloat16 values include:\n\n\\[\\begin{aligned} \\text{smallest positive normal} &=2^{-126} \\approx 1.17549\\times10^{-38},\\\\ \\text{smallest positive subnormal} &=2^{-133} \\approx 9.18355\\times10^{-41},\\\\ \\text{largest finite value} &=(2-2^{-7})2^{127} \\approx 3.38953\\times10^{38}. \\end{aligned}\\]\n\nThe gap between 1 and the next larger representable value is\n\n\\[\\epsilon=2^{-7}=0.0078125.\\]\n\nUnder rounding to nearest, the usual relative rounding-error bound for normal results away from overflow and underflow is \\(2^{-8}\\), or about 0.39%.\n\nSpacing grows with magnitude. Between 1 and 2, successive values are separated by \\(1/128\\); between 128 and 256, they are separated by 1. Every integer from \\(-256\\) through \\(256\\) is exactly representable, but not every integer beyond that interval is.\n\nBfloat16 and binary32 thus have the same normal exponent limits, but not identical sets of representable values. Binary32 has a slightly larger maximum finite value and supports substantially smaller subnormal values."
    },
    {
      "heading": "Conversion and arithmetic",
      "text": "Relationship to binary32\n\nBfloat16’s layout matches the most significant 16 bits of a binary32 encoding: the sign bit, all eight exponent bits, and the first seven fraction bits.\n\nAt the representation level, a bfloat16 value can be expanded to binary32 by appending 16 zero fraction bits. All finite bfloat16 values are exactly representable in binary32.\n\nThe reverse conversion generally loses information. Simply discarding the lower 16 bits truncates the significand; correctly rounded conversion may instead increment the retained portion according to a rounding rule. Common choices include round to nearest, ties to even, although the supported modes depend on the instruction or software implementation.\n\nConversion also needs appropriate handling of special values. In particular, NaN conversion must preserve a NaN encoding rather than accidentally producing infinity.\n\nStorage precision versus accumulation precision\n\nThe datatype used to store operands need not be the datatype used to accumulate results. In mixed-precision computing, bfloat16 inputs are frequently multiplied and accumulated into FP32 results. This is especially important for matrix multiplication and dot products, which add many products:\n\n\\[c_{ij}=\\sum_k a_{ik}b_{kj}.\\]\n\nUsing an FP32 accumulator reduces the rounding error that would result from repeatedly rounding the running sum to bfloat16. However, it cannot recover information already lost when the inputs were converted to bfloat16. Hardware may also differ in its treatment of intermediate products, grouping of additions, subnormals, and exceptional values.\n\nBfloat16 follows several conventions familiar from IEEE 754, including biased exponents and encodings for infinities and NaNs. It is nevertheless distinct from IEEE binary16, and the name alone does not specify every detail of an implementation’s arithmetic behavior."
    },
    {
      "heading": "Use in machine learning",
      "text": "Training\n\nDeep learning often relies on large matrix operations involving model weights, activations, and gradients. Many such workloads tolerate reduced significand precision in parts of the computation, while still requiring a broad range of magnitudes.\n\nBfloat16 offers several practical advantages:\n\n- Reduced storage: an array occupies half the raw storage of an equivalent FP32 array.\n- Reduced memory traffic: moving two-byte elements instead of four-byte elements can reduce bandwidth requirements.\n- Efficient arithmetic hardware: a shorter significand can reduce multiplier complexity and support greater arithmetic throughput.\n- Broad exponent range: small gradients and large activations are less likely to exceed its normal exponent range than with binary16.\n\nThese properties are useful during backpropagation, but they do not make every operation equally suitable for reduced precision. Training systems often retain higher precision for optimizer state, selected reductions, normalization calculations, or parameter updates.\n\nFor example, a small update may disappear if a weight and its updated value round to the same bfloat16 number. Keeping an FP32 master copy of weights is one way some mixed-precision training systems address this problem; the exact arrangement varies by framework and optimizer.\n\nBinary16 training commonly uses loss scaling, which rescales a loss function to help prevent small gradients from underflowing. Because bfloat16 has a much wider exponent range, this particular technique is generally unnecessary for bfloat16 training. Its broad range does not, however, prevent all numerical failures.\n\nInference\n\nBfloat16 is also used for neural-network inference, including large language models. Model parameters and intermediate activations stored in bfloat16 require less memory than FP32 equivalents.\n\nBfloat16 is distinct from integer-based neural-network quantization. It retains a floating-point exponent for each value and does not inherently require a separately selected scale for a tensor or group of values. It also normally requires more storage than 8-bit or 4-bit representations.\n\nThe precision used for weights, intermediate computations, accumulators, and outputs can differ within a single model execution."
    },
    {
      "heading": "Development and hardware support",
      "text": "The name bfloat16 derives from Brain floating point, reflecting its origin in Google’s Brain research work. It became prominent through Google’s Tensor Processing Units, particularly their use of bfloat16 multiplication with FP32 accumulation.\n\nSupport subsequently expanded to central processing units and graphics processing units. Examples include Intel processors with AVX-512 BF16 or appropriate Advanced Matrix Extensions support, and NVIDIA GPUs with suitable Tensor Core support. Availability and throughput depend on the particular processor generation and instruction set.\n\nFrameworks such as TensorFlow and PyTorch expose bfloat16 datatypes and mixed-precision facilities. Software support alone does not guarantee acceleration: unsupported operations may require conversion, execution in another precision, or emulation."
    },
    {
      "heading": "Limitations",
      "text": "Bfloat16’s short significand makes it unsuitable as a universal replacement for FP32 or FP64. Its main numerical limitations include:\n\n- Coarse resolution: nearby values may round to the same representation.\n- Loss of small updates: adding a small quantity to a much larger one may leave the stored result unchanged.\n- Cancellation: subtracting nearly equal values can expose errors introduced by earlier rounding.\n- Accumulation error: long sums can lose substantial accuracy without higher-precision accumulation.\n- Implementation differences: rounding, subnormal handling, and operation ordering can affect results across devices.\n\nSensitive scientific calculations, ill-conditioned problems, and computations requiring tight error bounds may need substantially more precision. Even in machine learning, numerical suitability depends on the model, operations, training configuration, and hardware.\n\nA twofold reduction in element storage also does not imply a twofold improvement in application speed or total memory use. Actual benefits depend on hardware throughput, memory bandwidth, conversion overhead, and the proportion of data that remains in higher precision."
    }
  ],
  "links": [
    "bit",
    "floating-point-arithmetic",
    "machine-learning",
    "google",
    "not-a-number",
    "matrix",
    "ieee-seven-five-four",
    "deep-learning",
    "backpropagation",
    "loss-function",
    "neural-network-inference",
    "large-language-model",
    "neural-network-quantization",
    "tensor-processing-unit",
    "central-processing-unit",
    "graphics-processing-unit",
    "tensorflow",
    "pytorch"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}