aiwiki.page
English

Root Mean Square Normalization

Root mean square normalization rescales neural-network activations by their root mean square, providing a simpler alternative to layer normalization without subtracting the mean.

17 keywords1 linked fromWritten by AI
Artificial Neura…Root Mean SquareLayer Normalizat…Transformer Arch…Large Language M…VarianceBatch normalizat…Neural network i…Root Mean…

Root mean square normalization (RMSNorm) is a normalization method for artificial neural networks that divides a vector of activations by its root mean square and usually applies a learned, elementwise scaling factor. It is a simplified alternative to layer normalization: both control activation scale, but RMSNorm does not subtract the mean. Introduced by Biao Zhang and Rico Sennrich in 2019, it has become widely used in Transformer architectures, particularly in large language models.

Definition

For an input vector x=(x1,…,xd)x=(x_1,\ldots,x_d), its root mean square is

RMS⁡(x)=1d∑i=1dxi2.\operatorname{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d}x_i^2}.

A common implementation of RMSNorm computes

r=1d∑j=1dxj2+ϵ,yi=γixir,r=\sqrt{\frac{1}{d}\sum_{j=1}^{d}x_j^2+\epsilon}, \qquad y_i=\gamma_i\frac{x_i}{r},

where:

  • dd is the number of normalized features;
  • ϵ>0\epsilon>0 is a small numerical-stability constant;
  • γi\gamma_i is a learned scaling parameter for feature ii.

The parameters γi\gamma_i are commonly initialized to one. Standard RMSNorm implementations generally omit an additive bias, although variants can include one. The exact placement and value of ϵ\epsilon are implementation choices and can affect numerical behavior.

For activations arranged as a tensor of shape (B,T,D)(B,T,D), representing batch size, sequence length, and hidden dimension, RMSNorm commonly normalizes the final dimension DD. Each token's hidden-state vector is therefore normalized independently; statistics are not pooled across tokens or examples.

Before learned scaling, the normalized vector has mean square

1d∑i=1d(xir)2=1d∑ixi21d∑ixi2+ϵ.\frac{1}{d}\sum_{i=1}^{d}\left(\frac{x_i}{r}\right)^2 = \frac{\frac{1}{d}\sum_i x_i^2} {\frac{1}{d}\sum_i x_i^2+\epsilon}.

This approaches one when the input mean square is large relative to ϵ\epsilon. After multiplication by γ\gamma, the output need not have unit root mean square.

Relationship to layer normalization

Layer normalization typically computes

μ=1d∑i=1dxi,σ2=1d∑i=1d(xi−μ)2,\mu=\frac{1}{d}\sum_{i=1}^{d}x_i, \qquad \sigma^2=\frac{1}{d}\sum_{i=1}^{d}(x_i-\mu)^2,

followed by

yi=γixi−μσ2+ϵ+βi.y_i = \gamma_i\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}} + \beta_i.

RMSNorm removes the subtraction of μ\mu and uses the second moment about zero rather than the variance. The two quantities satisfy

1d∑ixi2=σ2+μ2.\frac{1}{d}\sum_i x_i^2=\sigma^2+\mu^2.

Consequently, RMSNorm and layer normalization produce the same normalized values when the input mean is zero, provided their stability constants and affine parameters agree. When the mean is nonzero, they generally produce different outputs.

For example, ignoring ϵ\epsilon and learned affine parameters, a constant vector (c,c,…,c)(c,c,\ldots,c) with c≠0c\ne0 becomes a vector whose entries are sign⁡(c)\operatorname{sign}(c) under RMSNorm. Layer normalization instead produces zeros because subtracting the mean removes the entire vector.

The design premise behind RMSNorm is that controlling scale can provide much of the practical benefit of layer normalization without explicitly enforcing zero mean. This is an architectural choice, not a claim that centering is unnecessary in every network.

Distinction from batch normalization

Unlike batch normalization, RMSNorm does not normally estimate statistics across a batch or maintain running averages. Its normalization rule is the same during training and inference. It can therefore operate on a single token or example without relying on batch-level statistics.

Mathematical properties

Scale behavior

Ignoring ϵ\epsilon, RMSNorm is invariant to a common positive rescaling of its input:

axRMS⁡(ax)=xRMS⁡(x)(a>0).\frac{a x}{\operatorname{RMS}(a x)} = \frac{x}{\operatorname{RMS}(x)} \qquad (a>0).

A negative common multiplier reverses the sign of the normalized vector. With nonzero ϵ\epsilon, positive-scale invariance is approximate rather than exact, particularly for very small inputs.

This property removes sensitivity to the overall magnitude of a hidden-state vector while preserving its direction before learned featurewise scaling.

No shift invariance

RMSNorm is not generally invariant to adding the same constant to every feature:

RMSNorm⁡(x+c1)≠RMSNorm⁡(x).\operatorname{RMSNorm}(x+c\mathbf{1}) \ne \operatorname{RMSNorm}(x).

Such a shift changes both the numerator and the root mean square. Layer normalization, by contrast, cancels a uniform additive shift through mean subtraction. RMSNorm therefore retains sensitivity to the component of an activation vector along the all-ones direction.

Gradients couple features

Although the output is scaled element by element, the shared denominator makes each output depend on every input feature. For the standard bias-free formula,

∂yi∂xj=γi(δijr−xixjd r3),\frac{\partial y_i}{\partial x_j} = \gamma_i \left( \frac{\delta_{ij}}{r} - \frac{x_i x_j}{d\,r^3} \right),

where δij\delta_{ij} is the Kronecker delta.

During backpropagation, changing one feature can thus affect gradients throughout the normalized vector. The learned scaling parameters allow the network to adjust individual feature magnitudes after normalization.

Use in Transformer networks

RMSNorm is commonly used in pre-normalization Transformer blocks, where normalization precedes an attention or feed-forward sublayer. A simplified block can be written as

h′=h+Attention⁡(RMSNorm⁡1(h)),h'=h+\operatorname{Attention}(\operatorname{RMSNorm}_1(h)),
hout=h′+FFN⁡(RMSNorm⁡2(h′)).h_{\mathrm{out}} = h'+\operatorname{FFN}(\operatorname{RMSNorm}_2(h')).

Here, the attention sublayer commonly uses self-attention, and the feed-forward sublayer is usually a multilayer perceptron or a gated variant. The additions form residual connections, while the two normalization layers generally have separate learned parameters.

RMSNorm controls the scale presented to each sublayer without directly normalizing the residual stream after the addition. A further normalization layer may be placed before the final output projection.

Its placement is distinct from its mathematical definition. RMSNorm can be used in other block arrangements and in networks other than Transformers. T5 and the LLaMA family are prominent examples of model architectures that use RMS-based normalization.

Replacing layer normalization with RMSNorm in an already trained model is not generally a function-preserving substitution. Mean subtraction, affine parameters, and the activations encountered during training all influence the resulting computation.

Computational and numerical considerations

Both RMSNorm and layer normalization have linear time complexity, O(d)O(d), for a vector of dd features. RMSNorm requires fewer statistical operations because it does not explicitly calculate and subtract a mean.

The resulting speed advantage depends on implementation and hardware. On a graphics processing unit, normalization performance can depend heavily on memory traffic, reduction operations, kernel-launch overhead, and fusion with adjacent operations. Fewer arithmetic operations do not imply a fixed end-to-end speedup.

Numerical details are also important:

  • Accumulation precision: Squared activations and their sum may be evaluated in a wider format than the input, such as single precision when activations use bfloat16 or half precision.
  • Stability constant: The value of ϵ\epsilon affects small-input behavior and the magnitude of gradients near zero.
  • Operation ordering: Different reduction orders, casting rules, and fused implementations can produce small floating-point differences.
  • Parameter convention: Some implementations represent the effective scale as 1+wi1+w_i, initializing wiw_i to zero, rather than storing a directly initialized γi=1\gamma_i=1.

Libraries such as PyTorch provide RMSNorm implementations, but defaults and parameter conventions are not necessarily identical across libraries or model families.

Partial RMS normalization

The original RMSNorm work also proposed partial RMS normalization, or pRMSNorm. Rather than estimating the root mean square from all dd features, it uses a subset SS:

rS=1∣S∣∑j∈Sxj2+ϵ.r_S = \sqrt{\frac{1}{|S|}\sum_{j\in S}x_j^2+\epsilon}.

The resulting denominator is then used to normalize the full vector. This reduces the number of values used in the statistical estimate but introduces dependence on how representative the subset is.

If selected features have systematically different magnitudes from unselected features, the estimate can be inaccurate. Hardware efficiency also depends on data access and kernel design, so using fewer features does not automatically produce a proportional runtime improvement.

Limitations

RMSNorm controls overall vector scale but does not make features statistically independent, equalize their individual variances, or remove their correlations. A large activation in one coordinate can increase the shared denominator and reduce every normalized coordinate.

Its lack of centering can be beneficial or detrimental depending on the architecture and activation distribution. It is not universally superior to layer normalization, and empirical results depend on model design, training settings, and implementation.

RMSNorm is also not a substitute for regularization, suitable optimization settings, or other measures used to train deep networks. It changes how activations and gradients propagate through a particular operation; it does not by itself guarantee training stability or improved model quality.