{
  "id": "rms-normalization",
  "title": "Root Mean Square Normalization",
  "lang": "en",
  "source_lang": "en",
  "summary": "Root mean square normalization rescales neural-network activations by their root mean square, providing a simpler alternative to layer normalization without subtracting the mean.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "RMSNorm",
    "RMS normalization"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/rms-normalization/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/rms-normalization/",
      "json": "https://aiwiki.page/zh/rms-normalization.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Root mean square normalization (RMSNorm) is a normalization method for artificial neural networks that divides a vector of activations by its root mean square and usually applies a learned, elementwise scaling factor. It is a simplified alternative to layer normalization: both control activation scale, but RMSNorm does not subtract the mean. Introduced by Biao Zhang and Rico Sennrich in 2019, it has become widely used in Transformer architectures, particularly in large language models."
    },
    {
      "heading": "Definition",
      "text": "For an input vector \\(x=(x_1,\\ldots,x_d)\\), its root mean square is\n\n\\[\\operatorname{RMS}(x) = \\sqrt{\\frac{1}{d}\\sum_{i=1}^{d}x_i^2}.\\]\n\nA common implementation of RMSNorm computes\n\n\\[r=\\sqrt{\\frac{1}{d}\\sum_{j=1}^{d}x_j^2+\\epsilon}, \\qquad y_i=\\gamma_i\\frac{x_i}{r},\\]\n\nwhere:\n\n- \\(d\\) is the number of normalized features;\n- \\(\\epsilon>0\\) is a small numerical-stability constant;\n- \\(\\gamma_i\\) is a learned scaling parameter for feature \\(i\\).\n\nThe parameters \\(\\gamma_i\\) are commonly initialized to one. Standard RMSNorm implementations generally omit an additive bias, although variants can include one. The exact placement and value of \\(\\epsilon\\) are implementation choices and can affect numerical behavior.\n\nFor activations arranged as a tensor of shape \\((B,T,D)\\), representing batch size, sequence length, and hidden dimension, RMSNorm commonly normalizes the final dimension \\(D\\). Each token's hidden-state vector is therefore normalized independently; statistics are not pooled across tokens or examples.\n\nBefore learned scaling, the normalized vector has mean square\n\n\\[\\frac{1}{d}\\sum_{i=1}^{d}\\left(\\frac{x_i}{r}\\right)^2 = \\frac{\\frac{1}{d}\\sum_i x_i^2} {\\frac{1}{d}\\sum_i x_i^2+\\epsilon}.\\]\n\nThis approaches one when the input mean square is large relative to \\(\\epsilon\\). After multiplication by \\(\\gamma\\), the output need not have unit root mean square."
    },
    {
      "heading": "Relationship to layer normalization",
      "text": "Layer normalization typically computes\n\n\\[\\mu=\\frac{1}{d}\\sum_{i=1}^{d}x_i, \\qquad \\sigma^2=\\frac{1}{d}\\sum_{i=1}^{d}(x_i-\\mu)^2,\\]\n\nfollowed by\n\n\\[y_i = \\gamma_i\\frac{x_i-\\mu}{\\sqrt{\\sigma^2+\\epsilon}} + \\beta_i.\\]\n\nRMSNorm removes the subtraction of \\(\\mu\\) and uses the second moment about zero rather than the variance. The two quantities satisfy\n\n\\[\\frac{1}{d}\\sum_i x_i^2=\\sigma^2+\\mu^2.\\]\n\nConsequently, RMSNorm and layer normalization produce the same normalized values when the input mean is zero, provided their stability constants and affine parameters agree. When the mean is nonzero, they generally produce different outputs.\n\nFor example, ignoring \\(\\epsilon\\) and learned affine parameters, a constant vector \\((c,c,\\ldots,c)\\) with \\(c\\ne0\\) becomes a vector whose entries are \\(\\operatorname{sign}(c)\\) under RMSNorm. Layer normalization instead produces zeros because subtracting the mean removes the entire vector.\n\nThe design premise behind RMSNorm is that controlling scale can provide much of the practical benefit of layer normalization without explicitly enforcing zero mean. This is an architectural choice, not a claim that centering is unnecessary in every network.\n\nDistinction from batch normalization\n\nUnlike batch normalization, RMSNorm does not normally estimate statistics across a batch or maintain running averages. Its normalization rule is the same during training and inference. It can therefore operate on a single token or example without relying on batch-level statistics."
    },
    {
      "heading": "Mathematical properties",
      "text": "Scale behavior\n\nIgnoring \\(\\epsilon\\), RMSNorm is invariant to a common positive rescaling of its input:\n\n\\[\\frac{a x}{\\operatorname{RMS}(a x)} = \\frac{x}{\\operatorname{RMS}(x)} \\qquad (a>0).\\]\n\nA negative common multiplier reverses the sign of the normalized vector. With nonzero \\(\\epsilon\\), positive-scale invariance is approximate rather than exact, particularly for very small inputs.\n\nThis property removes sensitivity to the overall magnitude of a hidden-state vector while preserving its direction before learned featurewise scaling.\n\nNo shift invariance\n\nRMSNorm is not generally invariant to adding the same constant to every feature:\n\n\\[\\operatorname{RMSNorm}(x+c\\mathbf{1}) \\ne \\operatorname{RMSNorm}(x).\\]\n\nSuch a shift changes both the numerator and the root mean square. Layer normalization, by contrast, cancels a uniform additive shift through mean subtraction. RMSNorm therefore retains sensitivity to the component of an activation vector along the all-ones direction.\n\nGradients couple features\n\nAlthough the output is scaled element by element, the shared denominator makes each output depend on every input feature. For the standard bias-free formula,\n\n\\[\\frac{\\partial y_i}{\\partial x_j} = \\gamma_i \\left( \\frac{\\delta_{ij}}{r} - \\frac{x_i x_j}{d\\,r^3} \\right),\\]\n\nwhere \\(\\delta_{ij}\\) is the Kronecker delta.\n\nDuring backpropagation, changing one feature can thus affect gradients throughout the normalized vector. The learned scaling parameters allow the network to adjust individual feature magnitudes after normalization."
    },
    {
      "heading": "Use in Transformer networks",
      "text": "RMSNorm is commonly used in pre-normalization Transformer blocks, where normalization precedes an attention or feed-forward sublayer. A simplified block can be written as\n\n\\[h'=h+\\operatorname{Attention}(\\operatorname{RMSNorm}_1(h)),\\]\n\n\\[h_{\\mathrm{out}} = h'+\\operatorname{FFN}(\\operatorname{RMSNorm}_2(h')).\\]\n\nHere, the attention sublayer commonly uses self-attention, and the feed-forward sublayer is usually a multilayer perceptron or a gated variant. The additions form residual connections, while the two normalization layers generally have separate learned parameters.\n\nRMSNorm controls the scale presented to each sublayer without directly normalizing the residual stream after the addition. A further normalization layer may be placed before the final output projection.\n\nIts placement is distinct from its mathematical definition. RMSNorm can be used in other block arrangements and in networks other than Transformers. T5 and the LLaMA family are prominent examples of model architectures that use RMS-based normalization.\n\nReplacing layer normalization with RMSNorm in an already trained model is not generally a function-preserving substitution. Mean subtraction, affine parameters, and the activations encountered during training all influence the resulting computation."
    },
    {
      "heading": "Computational and numerical considerations",
      "text": "Both RMSNorm and layer normalization have linear time complexity, \\(O(d)\\), for a vector of \\(d\\) features. RMSNorm requires fewer statistical operations because it does not explicitly calculate and subtract a mean.\n\nThe resulting speed advantage depends on implementation and hardware. On a graphics processing unit, normalization performance can depend heavily on memory traffic, reduction operations, kernel-launch overhead, and fusion with adjacent operations. Fewer arithmetic operations do not imply a fixed end-to-end speedup.\n\nNumerical details are also important:\n\n- Accumulation precision: Squared activations and their sum may be evaluated in a wider format than the input, such as single precision when activations use bfloat16 or half precision.\n- Stability constant: The value of \\(\\epsilon\\) affects small-input behavior and the magnitude of gradients near zero.\n- Operation ordering: Different reduction orders, casting rules, and fused implementations can produce small floating-point differences.\n- Parameter convention: Some implementations represent the effective scale as \\(1+w_i\\), initializing \\(w_i\\) to zero, rather than storing a directly initialized \\(\\gamma_i=1\\).\n\nLibraries such as PyTorch provide RMSNorm implementations, but defaults and parameter conventions are not necessarily identical across libraries or model families."
    },
    {
      "heading": "Partial RMS normalization",
      "text": "The original RMSNorm work also proposed partial RMS normalization, or pRMSNorm. Rather than estimating the root mean square from all \\(d\\) features, it uses a subset \\(S\\):\n\n\\[r_S = \\sqrt{\\frac{1}{|S|}\\sum_{j\\in S}x_j^2+\\epsilon}.\\]\n\nThe resulting denominator is then used to normalize the full vector. This reduces the number of values used in the statistical estimate but introduces dependence on how representative the subset is.\n\nIf selected features have systematically different magnitudes from unselected features, the estimate can be inaccurate. Hardware efficiency also depends on data access and kernel design, so using fewer features does not automatically produce a proportional runtime improvement."
    },
    {
      "heading": "Limitations",
      "text": "RMSNorm controls overall vector scale but does not make features statistically independent, equalize their individual variances, or remove their correlations. A large activation in one coordinate can increase the shared denominator and reduce every normalized coordinate.\n\nIts lack of centering can be beneficial or detrimental depending on the architecture and activation distribution. It is not universally superior to layer normalization, and empirical results depend on model design, training settings, and implementation.\n\nRMSNorm is also not a substitute for regularization, suitable optimization settings, or other measures used to train deep networks. It changes how activations and gradients propagate through a particular operation; it does not by itself guarantee training stability or improved model quality."
    }
  ],
  "links": [
    "artificial-neural-network",
    "root-mean-square",
    "layer-normalization",
    "transformer-architecture",
    "large-language-model",
    "variance",
    "batch-normalization",
    "neural-network-inference",
    "backpropagation",
    "self-attention",
    "multilayer-perceptron",
    "residual-connection",
    "time-complexity",
    "graphics-processing-unit",
    "bfloat16",
    "pytorch",
    "regularization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}