Rotary position embeddings (RoPE), from first principles

Rotary position embedding was introduced by Su et al. in RoFormer in 2021, and within two years it was the position scheme of nearly every open language model: Llama, Mistral, Qwen, Gemma and DeepSeek all use it. It has no parameters and adds a few multiply-adds per element. The idea fits in one sentence: rotate each query and key by an angle proportional to its position, so that the angle between a query and a key, and therefore their dot product, depends only on how far apart they are.

The worked numbers use a head of width $d = 128$ with base 10,000 and a 4,096-token context, the configuration of Llama 2 7B.

Attention needs to be told where tokens are

Attention scores every pair of positions by $q_m \cdot k_n$ and averages values by those scores. Nothing in that computation refers to $m$ or $n$: shuffle the input tokens and the outputs shuffle the same way. "Dog bites man" and "man bites dog" would get the same set of vectors. Position has to be put in.

Absolute schemes give each position its own vector and add it to the token embedding. The original Transformer used fixed sinusoids, $\sin(m / 10000^{2i/d})$ and $\cos(m / 10000^{2i/d})$ for $i = 0, \dots, d/2 - 1$; BERT and GPT-2 learned a table with one row per position, 512 and 1,024 rows. A learned table has nothing to offer past its last row. And an added position leaks into everything: with $x_m + p_m$ in place of $x_m$, the score $(x_m + p_m)^{\top} W_Q^{\top} W_K (x_n + p_n)$ expands into four terms, content with content, content with position, position with content and position with position, and the model has to learn to extract "how far apart" from absolute pieces.

Relative schemes put the offset $m - n$ into the score directly. Shaw et al. learned an embedding per clipped distance; T5 adds a learned scalar per head and distance bucket to each logit; ALiBi subtracts a fixed slope times the distance. These change the score by a term that depends on the pair of positions, added inside the attention computation.

RoPE asks for both at once: encode each position absolutely, on its own query or key, in a way that makes the score relative. Formally, find functions with

Two dimensions: a rotation

Take a query and a key with two components each and read them as complex numbers, $q = q_1 + i q_2$ and $k = k_1 + i k_2$. Encode position $m$ by multiplying by $e^{i m \theta}$, a rotation by the angle $m\theta$. The dot product of two plane vectors $a$ and $b$ is $\text{Re}[a \bar{b}]$, so

The absolute angles $m\theta$ and $n\theta$ have cancelled; only their difference is left. In matrix form, with $R(\alpha)$ the 2 × 2 rotation by $\alpha$, the same fact is two identities, $R(\alpha)^{\top} = R(-\alpha)$ and $R(\alpha) R(\beta) = R(\alpha + \beta)$:

Rotations also preserve length, so RoPE changes only the angle between a query and a key, never their norms.

All $d$ dimensions: $d/2$ clocks

A head has $d$ dimensions, not two. RoPE splits them into $d/2$ pairs and rotates pair $i$ at its own frequency,

with base 10,000 in the paper: the same frequencies as the 2017 sinusoids. The full rotation $R_m$ is block-diagonal, one 2 × 2 rotation by $m\theta_i$ per pair, so every pair obeys the identity above on its own, and

Nobody builds the matrix. For each pair, $(x_{2i}, x_{2i+1})$ becomes $(x_{2i} \cos m\theta_i - x_{2i+1} \sin m\theta_i,\; x_{2i} \sin m\theta_i + x_{2i+1} \cos m\theta_i)$: two multiplies and an add per element, with the cosines and sines looked up from a table indexed by position. Which dimensions form a pair is a convention. RoFormer pairs neighbours; the Hugging Face Llama code pairs dimension $j$ with $j + d/2$, so the conversion script permutes the rows of $W_Q$ and $W_K$. Both are correct; mixing them is a classic bug.

Three details matter in practice. RoPE is applied to queries and keys only, never to values, because only the score needs position. It is applied in every layer, after the projections, not once at the input, so the residual stream itself carries no position vector. And the KV cache stores keys already rotated, which is why changing the rotation after the fact means recomputing the cache.

What base 10,000 means

Pair $i$ turns by $\theta_i$ radians per token, so it completes a full turn every $2\pi / \theta_i$ tokens. With $d = 128$:

The fast pairs spin through many turns over a context and resolve fine differences in position; the slow ones barely move, and over a 4,096-token context the slowest turns through only 0.47 radians. That spread is RoPE's clock face: a short hand and a long hand, and 62 in between. The paper also shows that with this choice of frequencies an upper bound on the score decays as $|m - n|$ grows, a mild built-in preference for nearby tokens.

The spread also explains why RoPE fails past its training length. With base 10,000 and 4,096 training tokens, 18 of the 64 pairs have a wavelength longer than 4,096 tokens: during training they never completed a turn, so the model has never seen them at the angles that offsets beyond 4,096 produce. Run the model at 8,000 tokens and those pairs show it unfamiliar angles, and attention degrades. Every long-context method below is a way of avoiding those angles.

Parameters and FLOPs

RoPE has no parameters. Its cost is the rotation of $q$ and $k$: in Llama 2 7B, three FLOPs for each of 4,096 query and 4,096 key elements, 24,576 FLOPs per token per layer, against 134,217,728 for the attention's four weight matrices. It is negligible next to the matrix multiplies. The one real cost is conceptual: because the rotation sits between the key projection and the cache, anything that wants to compress or share keys has to work around it (below).

Longer than training: four ways to stretch

Say a model was trained on $L$ tokens and should run on $sL$.

• Position interpolation (Chen et al., 2023). Divide every position by $s$: position $m$ is rotated as if it were $m/s$, so the whole longer context maps into the trained range, at the price of crowding neighbouring positions together on every pair, the fast ones included. With fine-tuning of at most 1,000 steps it extended Llama models to 32,768 tokens.
• NTK-aware scaling (first posted by a user, bloc97, on Reddit in 2023). Leave positions alone and raise the base to $\text{base} \cdot s^{d/(d-2)}$. Then $\theta_i$ is multiplied by $s^{-2i/(d-2)}$: the fastest pair is untouched and the slowest is slowed by exactly $s$. For $s = 4$ with $d = 128$, the base goes from 10,000 to about 40,890. Fine position resolution is kept, and only the slow pairs are interpolated.
• YaRN (Peng et al., 2023) makes that split explicit, per pair. Count how many turns pair $i$ makes in the training context, $r_i = L\theta_i / 2\pi$. Pairs with $r_i$ below 1 are fully interpolated, divided by $s$; pairs above 32 are left alone; those in between are blended linearly. For Llama 2 at 4,096 tokens that is 18 pairs interpolated, 21 untouched and 25 blended. YaRN also scales the logits by $(0.1 \ln s + 1)^2$, a temperature the authors found lowers perplexity evenly across the extended window. The paper reports reaching its results with 10 times fewer tokens and 2.5 times fewer training steps than earlier methods, and extended Llama 2 to 128K tokens.
• Raise the base and keep training. Rather than rescale a finished model, pick a larger base and train with it, on long sequences for at least part of training. Code Llama moved from 10,000 to 1,000,000 (Rozière et al., 2023); Llama 3 uses 500,000 (Llama 3 paper), where the slowest of 64 pairs needs 2,559,196 tokens per turn. Llama 3.1 then reached 128K tokens with a per-pair interpolation rule of the YaRN kind on top.

Partial RoPE

Nothing requires every pair to rotate. GPT-J and GPT-NeoX-20B rotate only the first 25% of each head's dimensions and leave the rest as plain dot products: a channel that matches content at any distance, at a quarter of the cost. Barbero et al. (2024) found that Gemma uses the highest frequencies for positional heads and prefers the lowest ones, which barely move over a context and which the authors suspect carry semantic information, and proposed p-RoPE, which removes the lowest frequencies entirely. Gemma 4 uses pp-RoPE on its global layers with a base of 1M, and plain RoPE with base 10k on its local ones.

Partial RoPE also solves a structural problem. Multi-head latent attention caches one compressed latent per token and folds the key up-projection into the query, which is impossible if a position-dependent rotation sits between them. DeepSeek-V2 therefore keeps the compressed part unrotated and carries position on a separate small rotated key of 64 dimensions, shared by all heads. And cross-model KV transfer has to undo one model's rotation before mapping its keys into another's.

What changed since 2021

The mechanism has not changed at all; the frequencies have. Bases went from 10,000 to hundreds of thousands and millions, contexts from 2,048 tokens to more than 128K, and per-pair rules replaced uniform scaling. Multimodal models split the pairs between axes, as in Qwen2-VL's M-RoPE, with some pairs for time, some for image height and some for width. Encoders adopted it too: ModernBERT uses base 160,000 on its global layers. And GRAPE shows that RoPE is one case of a general construction, a position acting through a group, of which ALiBi is another.

What RoPE is good at is in the proof: relative position with no parameters, no change to the attention kernel, and no cost worth counting. What it is bad at is the table above: a model knows only the angles it was trained on, and every method for going longer is a way of never showing it new ones.