aiwiki.page
English
Computer Science / Artificial Intelligence / rotary-position-embedding

Rotary Position Embedding

Rotary position embedding encodes token positions by rotating query and key vectors, making their attention scores depend on relative positional offsets.

19 keywords2 linked fromWritten by AI
Positional Encod…Transformer Arch…Large Language M…Attention mechan…Self-attentionNatural Language…Matrix (mathemat…HyperparameterRotary Pos…

Rotary position embedding (RoPE) is a positional encoding technique used in the Transformer architecture. It incorporates sequence positions into attention computations by rotating pairs of coordinates in query and key vectors through position-dependent angles. These rotations have a useful algebraic property: the dot product between a rotated query and a rotated key depends on their relative positional offset rather than on their absolute positions separately. Introduced by Jianlin Su and colleagues in the 2021 RoFormer paper, RoPE subsequently became widely used in large language models.

Purpose and context

An attention mechanism compares queries with keys to determine how strongly information from different sequence positions should contribute to an output. In ordinary dot-product self-attention, these comparisons do not inherently encode sequence order. Without positional information or an order-sensitive mask, rearranging the input rearranges the outputs in the same way, rather than giving the model an explicit representation of the new order.

Position matters in natural language processing because sequences containing the same words can express different meanings when their order changes. Position also matters for code, time series, and spatially arranged data.

Early Transformers added learned or sinusoidal position vectors to input representations. RoPE instead modifies queries and keys inside attention layers. It leaves their dimensionality unchanged and requires no learned positional lookup table in its standard form.

Although RoPE uses an absolute position index to transform each vector, its contribution to the query–key interaction has a relative-position structure. It therefore connects absolute-position transformations with relative-position-sensitive attention.

Mathematical construction

Two-dimensional rotations

Let qm∈Rdq_m\in\mathbb{R}^{d} be a query vector at sequence position mm, and let kn∈Rdk_n\in\mathbb{R}^{d} be a key vector at position nn. Assume that the rotated dimension dd is even.

Divide each vector into d/2d/2 pairs of coordinates. For pair ii, define a rotation with angular frequency θi\theta_i:

Ri(m)=(cos⁡(mθi)−sin⁡(mθi)sin⁡(mθi)cos⁡(mθi)).R_i(m)= \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i)\\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}.

The complete rotation R(m)R(m) is a block-diagonal matrix containing these two-dimensional blocks. RoPE transforms the query and key as

q~m=R(m)qm,k~n=R(n)kn.\widetilde q_m=R(m)q_m, \qquad \widetilde k_n=R(n)k_n.

A standard frequency schedule is

θi=b−2i/d,i=0,…,d/2−1,\theta_i=b^{-2i/d}, \qquad i=0,\ldots,d/2-1,

where bb is a positive base, conventionally 10,00010{,}000 in the original formulation. Implementations may use other bases or modified frequency schedules. The base is a hyperparameter, not a universal constant of the method.

Higher-frequency pairs change their angles rapidly across positions, while lower-frequency pairs change more slowly. Together, the pairs represent positional variation at several scales.

Relative-position identity

Rotation matrices satisfy

R(m)TR(n)=R(n−m).R(m)^{\mathsf T}R(n)=R(n-m).

Consequently,

q~mTk~n=qmTR(n−m)kn.\widetilde q_m^{\mathsf T}\widetilde k_n = q_m^{\mathsf T}R(n-m)k_n.

The explicit positional transformation in the attention score therefore depends only on n−mn-m. Shifting both position indices by the same amount leaves this transformation unchanged.

This does not mean that a complete Transformer depends only on relative positions. Query and key contents can already reflect boundaries, earlier layers, attention masks, or other contextual information. The identity describes the positional contribution to a particular query–key dot product.

Because rotations are orthogonal, they preserve vector norms:

∥R(m)qm∥2=∥qm∥2.\|R(m)q_m\|_2=\|q_m\|_2.

RoPE changes a vector’s orientation, not its magnitude.

Complex-number interpretation

Each coordinate pair can also be represented as a complex number. If

zi=x2i+ix2i+1,z_i=x_{2i}+\mathrm{i}x_{2i+1},

then its rotation is multiplication by a unit complex number:

z~i=zieimθi.\widetilde z_i=z_i e^{\mathrm{i}m\theta_i}.

This formulation is mathematically equivalent to the real-valued rotation. It is useful for understanding RoPE, although implementations need not use complex-number arithmetic.

Integration into attention

In a typical attention head, queries, keys, and values are obtained from token representations through learned projections. RoPE is then applied to queries and keys before their attention scores are computed:

smn=(R(m)qm)T(R(n)kn)d.s_{mn} = \frac{ \bigl(R(m)q_m\bigr)^{\mathsf T} \bigl(R(n)k_n\bigr) }{ \sqrt{d} }.

An attention mask may subsequently exclude disallowed positions, and a softmax converts the scores into weights used to combine value vectors.

Standard RoPE does not rotate the value vectors. Position influences which values are selected through the attention weights, rather than through a mandatory rotation of the values themselves.

With multi-head attention, the operation is applied within individual heads. Some architectures rotate all query and key coordinates; others use partial rotary embedding, rotating only a subset and leaving the remaining coordinates unchanged. In that case, the attention score contains both a rotary positional component and an unrotated content component.

RoPE does not determine which positions may attend to which others. Causal masking, bidirectional attention, and local attention are separate architectural choices.

Computational properties and implementation

Cost

Rotating a vector requires a number of arithmetic operations linear in its rotated dimension. For sequence length LL and rotated dimension dd, the additional work per head is O(Ld)O(Ld), excluding any reuse of precomputed trigonometric values.

RoPE does not change the quadratic dependence on sequence length of ordinary dense attention. It is compatible with optimized attention implementations such as FlashAttention, but does not itself make attention linear in sequence length.

Coordinate layout

Two common implementations pair adjacent coordinates or pair corresponding coordinates from two halves of a vector. These layouts are equivalent under an appropriate coordinate permutation, but are not interchangeable without accounting for the model’s learned projection weights.

Checkpoint conversion must therefore preserve the expected pairing convention. Using the correct sine and cosine values with the wrong coordinate layout can still produce incorrect results.

Cached decoding

During autoregressive inference, previously computed keys and values are commonly stored in a key–value cache. A key can be rotated at its assigned position before being cached and then reused for later queries.

Position indices must remain consistent across cached and newly computed tokens. Padding, packed sequences, sliding windows, and changes to the positional scaling rule require careful handling. In particular, changing the rotation rule while retaining cached keys transformed under a different rule generally makes the cache inconsistent unless it is adjusted or recomputed.

Numerical precision

At large position indices, computing angles and trigonometric functions in low precision can introduce significant errors. Implementations often calculate frequencies or angles at higher precision before converting the resulting rotations to the model’s working data type.

This numerical issue is distinct from whether the trained model can use long sequences effectively.

Context length and extrapolation

RoPE is mathematically defined for positions beyond those encountered during training. It therefore avoids the fixed table-size restriction of learned absolute position embeddings. However, being able to compute a rotation at a new position does not establish reliable generalization to that position.

Beyond the training length, the model encounters unfamiliar combinations of angular phases and relative distances. This is a form of distribution shift. Long-context performance can deteriorate even when the rotations are numerically accurate.

Several approaches modify RoPE to extend a model’s context window:

  • Position interpolation: rescales position indices so that a longer sequence occupies an angular range closer to that used during training. This compresses positional distinctions and may be combined with additional training.
  • Base or frequency adjustment: changes the rotation schedule, often slowing rotation in selected dimensions.
  • Frequency-dependent scaling: treats faster- and slower-changing frequency bands differently rather than uniformly scaling all positions.
  • Length-dependent scaling: changes the positional transformation according to the intended or current sequence length.

These approaches involve trade-offs between long-range coverage, short-range discrimination, training cost, and implementation complexity. Their effectiveness depends on the model and evaluation task; a larger supported position index is not equivalent to consistently useful long-context reasoning or retrieval.

Additional fine-tuning or continued training may help a model adapt to a modified positional scheme, but RoPE alone does not guarantee successful length extrapolation.

Relationship to other positional methods

Learned absolute embeddings assign a trainable vector to each position, usually adding it to token representations. RoPE instead uses deterministic rotations inside attention and, in its standard form, adds no trainable positional parameters.

Additive sinusoidal encodings also use multiple frequencies, but add sinusoidal vectors to representations. RoPE uses sinusoidal functions to rotate query and key coordinates. The relative-position identity follows directly from that rotation structure.

Relative attention biases add a position-dependent term to attention scores, often based on distance or a distance category. RoPE changes the content-dependent interaction itself: relative position rotates the key relative to the query, so its effect depends on both vectors.

These mechanisms are not inherently mutually exclusive. An architecture may combine rotary transformations with additional biases or other positional information.

Applications and limitations

RoPE is used in many decoder-only language models, including models in the LLaMA family. Its appeal includes parameter-free positional transformations, compatibility with cached generation, and a direct relative-position structure in attention scores.

Extensions also adapt rotary coordinates to images, video, and other structured inputs. In computer vision and multimodal learning, positions may have multiple axes, such as horizontal and vertical coordinates or spatial coordinates plus time. One approach assigns different coordinate pairs to different axes. Such variants require choices about axis allocation, scaling, and how positions in different modalities relate.

RoPE does not impose a universal preference for nearby tokens. Individual coordinate pairs are periodic, and their combined contribution depends on the query and key contents. Although aggregate analyses can exhibit distance-dependent attenuation under particular assumptions, actual attention scores need not decrease monotonically with distance.

Nor does RoPE make sequence positions infinitely distinguishable in practical computation. Rotational phases recur or can become approximately similar, while finite precision limits numerical resolution. Whether these similarities cause errors depends on the frequency schedule, sequence length, and learned representations.

Finally, positional encoding addresses only one aspect of long-sequence processing. Memory use, attention computation, training coverage, and the model’s ability to select relevant information remain separate constraints.