GRAPE: RoPE, ALiBi, and FoX are the same construction

Self-attention has no idea what order its tokens came in — permute the sequence and the raw attention scores are unchanged. So every transformer bolts on a positional encoding, and over the years these have multiplied into a small zoo of unrelated-looking tricks. Rotary embeddings (RoPE) rotate queries and keys by angle-per-position. ALiBi subtracts a linear penalty proportional to how far apart two tokens are. The Forgetting Transformer (FoX) multiplies in a per-token forget gate. Each is derived on its own terms, with its own intuition, and the folklore treats them as competing families: multiplicative phase versus additive bias.

GRAPE — Group Representational Position Encoding, from Princeton, UCLA and Tsinghua's IIIS (ICLR 2026) — makes the deflationary claim that all of them are the same construction seen through different generators. A position $n$ acts on the query/key space through one group action,

$\mathbf{G}(n) = \exp(n\,\omega\,\mathbf{L}),$

and which kind of matrix you put in the generator $\mathbf{L}$ decides the family. A rank-2 skew-symmetric $\mathbf{L}$ gives a rotation, and RoPE falls out exactly. A rank-1 nilpotent $\mathbf{L}$ gives a shear that injects a linear bias, and ALiBi and FoX fall out exactly. Scrub a position below and flip the generator to watch the two behaviours emerge from one law:

The payoff of the unification is not a new record — it's a design space. Once RoPE is "the rotation with the canonical basis and a log-uniform spectrum," you can ask what the learned basis does; once ALiBi is "the rank-1 unipotent action with a fixed slope," you can ask what a content-dependent slope does. GRAPE names and tries both.

One law, two generators

The organizing principle is that a positional map should respect an exact relative law:

$\mathbf{G}(t-s) = \mathbf{G}(s)^{-1}\,\mathbf{G}(t), \qquad \mathbf{G}(n+m) = \mathbf{G}(n)\,\mathbf{G}(m).$

This is what makes attention translation-invariant: if you apply $\mathbf{G}(i)$ to query $i$ and $\mathbf{G}(j)$ to key $j$, the score $\tilde{\mathbf{q}}_i^\top\tilde{\mathbf{k}}_j = \mathbf{q}_i^\top\mathbf{G}(j-i)\mathbf{k}_j$ depends only on the offset $j-i$, never on absolute position. Any one-parameter subgroup $\mathbf{G}(n) = \exp(n\mathbf{L})$ satisfies it automatically — so the whole design question collapses to which generator $\mathbf{L}$ to exponentiate. GRAPE identifies the two generator types that keep the exponential cheap and the geometry clean:

• Rank-2 skew $\mathbf{L} = \mathbf{a}\mathbf{b}^\top - \mathbf{b}\mathbf{a}^\top \in \mathfrak{so}(d)$ exponentiates to an orthogonal map — a norm-preserving rotation in $\mathrm{SO}(d)$. This is Multiplicative GRAPE.
• Rank-1 nilpotent $\mathbf{A}$ with $\mathbf{A}^2 = \mathbf{0}$ exponentiates, in one term, to a unipotent map $\mathbf{G}(n) = \mathbf{I} + n\omega\mathbf{A}$ in the general linear group $\mathrm{GL}$ — a shear that translates a feature and shows up as an additive logit bias. This is Additive GRAPE.

expand

Multiplicative GRAPE: RoPE is a rotation with a fixed basis

Build the generator from two vectors $\mathbf{a},\mathbf{b}\in\mathbb{R}^d$. With $\alpha = \|\mathbf{a}\|^2$, $\beta = \|\mathbf{b}\|^2$, $\gamma = \mathbf{a}^\top\mathbf{b}$ and $s = \sqrt{\alpha\beta - \gamma^2}$, the rank-2 skew $\mathbf{L}$ squares to $\mathbf{L}^2 = -s^2\,\mathbf{P}_{\mathcal{U}}$ on the plane $\mathcal{U} = \mathrm{span}\{\mathbf{a},\mathbf{b}\}$. That single fact collapses the matrix exponential to a Rodrigues-type closed form:

$\exp(\mathbf{L}) = \mathbf{I} + \frac{\sin s}{s}\mathbf{L} + \frac{1-\cos s}{s^2}\mathbf{L}^2,$

a pure rotation by angle $s$ inside the plane $\mathcal{U}$, computable in $O(d)$ flops with no matrix ever materialized. Stack $d/2$ of these on disjoint coordinate pairs with frequencies $\theta_i$ and they commute, so