# GRAPE: RoPE, ALiBi, and FoX are the same construction

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/grape-position-encoding
> date: 2026-07-16
> tags: llm, attention, positional-encoding, transformers, explainer
Self-attention has no idea what order its tokens came in — permute the sequence and the raw
attention scores are unchanged. So every transformer bolts on a *positional encoding*, and over the
years these have multiplied into a small zoo of unrelated-looking tricks. [Rotary embeddings
(RoPE)](/articles/attention-mechanisms) rotate queries and keys by angle-per-position. **ALiBi**
subtracts a linear penalty proportional to how far apart two tokens are. The **Forgetting Transformer
(FoX)** multiplies in a per-token forget gate. Each is derived on its own terms, with its own
intuition, and the folklore treats them as competing families: multiplicative *phase* versus additive
*bias*.

**GRAPE** — *Group Representational Position Encoding*, from Princeton, UCLA and Tsinghua's IIIS
(ICLR 2026) — makes the deflationary claim that all of them are the **same construction seen through
different generators**. A position $n$ acts on the query/key space through one group action,

$$\mathbf{G}(n) = \exp(n\,\omega\,\mathbf{L}),$$

and *which kind of matrix* you put in the generator $\mathbf{L}$ decides the family. A rank-2
skew-symmetric $\mathbf{L}$ gives a **rotation**, and RoPE falls out exactly. A rank-1 nilpotent
$\mathbf{L}$ gives a **shear** that injects a linear bias, and ALiBi and FoX fall out exactly. Scrub a
position below and flip the generator to watch the two behaviours emerge from one law:

<GeneratorAction />

The payoff of the unification is not a new record — it's a *design space*. Once RoPE is "the rotation
with the canonical basis and a log-uniform spectrum," you can ask what the *learned* basis does; once
ALiBi is "the rank-1 unipotent action with a fixed slope," you can ask what a *content-dependent* slope
does. GRAPE names and tries both.

## One law, two generators

The organizing principle is that a positional map should respect an **exact relative law**:

$$\mathbf{G}(t-s) = \mathbf{G}(s)^{-1}\,\mathbf{G}(t), \qquad \mathbf{G}(n+m) = \mathbf{G}(n)\,\mathbf{G}(m).$$

This is what makes attention translation-invariant: if you apply $\mathbf{G}(i)$ to query $i$ and
$\mathbf{G}(j)$ to key $j$, the score $\tilde{\mathbf{q}}_i^\top\tilde{\mathbf{k}}_j =
\mathbf{q}_i^\top\mathbf{G}(j-i)\mathbf{k}_j$ depends only on the offset $j-i$, never on absolute
position. Any one-parameter subgroup $\mathbf{G}(n) = \exp(n\mathbf{L})$ satisfies it automatically —
so the whole design question collapses to *which generator $\mathbf{L}$ to exponentiate*. GRAPE
identifies the two generator types that keep the exponential cheap and the geometry clean:

- **Rank-2 skew** $\mathbf{L} = \mathbf{a}\mathbf{b}^\top - \mathbf{b}\mathbf{a}^\top \in
  \mathfrak{so}(d)$ exponentiates to an **orthogonal** map — a norm-preserving rotation in
  $\mathrm{SO}(d)$. This is *Multiplicative GRAPE*.
- **Rank-1 nilpotent** $\mathbf{A}$ with $\mathbf{A}^2 = \mathbf{0}$ exponentiates, in one term, to a
  **unipotent** map $\mathbf{G}(n) = \mathbf{I} + n\omega\mathbf{A}$ in the general linear group $\mathrm{GL}$
  — a shear that translates a feature and shows up as an additive logit bias. This is *Additive GRAPE*.

<Figure
  src="/articles/grape-position-encoding/fig1.png"
  alt="Overview diagram of the GRAPE framework. A top box states the general relative law G(t−s)=G(s)^{-1}G(t) and the map G(n)=exp(nω·Generator). Two arrows fork down to a blue Multiplicative GRAPE panel (Operation: Rotation, Manifold SO(d), rank-2 skew generator L=ab^T−ba^T with a Rodrigues closed form, a rotating-vector inset, recovers RoPE, extends to learned bases) and a red Additive GRAPE panel (Operation: Translation, Manifold GL(d+k) unipotent lift, rank-1 nilpotent generator A with A²=0 so exp(A)=I+A, a descending bias-vs-position inset, recovers ALiBi and FoX, extends to path integral)."
  caption="One framework, two generator types: a rank-2 skew generator gives a norm-preserving rotation (recovering RoPE); a rank-1 nilpotent generator gives a unipotent shear/additive bias (recovering ALiBi and FoX). Both obey the same relative law (Zhang et al., 2026, Figure 1)."
/>

## Multiplicative GRAPE: RoPE is a rotation with a fixed basis

Build the generator from two vectors $\mathbf{a},\mathbf{b}\in\mathbb{R}^d$. With
$\alpha = \|\mathbf{a}\|^2$, $\beta = \|\mathbf{b}\|^2$, $\gamma = \mathbf{a}^\top\mathbf{b}$ and
$s = \sqrt{\alpha\beta - \gamma^2}$, the rank-2 skew $\mathbf{L}$ squares to
$\mathbf{L}^2 = -s^2\,\mathbf{P}_{\mathcal{U}}$ on the plane $\mathcal{U} = \mathrm{span}\{\mathbf{a},\mathbf{b}\}$.
That single fact collapses the matrix exponential to a **Rodrigues-type closed form**:

$$\exp(\mathbf{L}) = \mathbf{I} + \frac{\sin s}{s}\mathbf{L} + \frac{1-\cos s}{s^2}\mathbf{L}^2,$$

a pure rotation by angle $s$ inside the plane $\mathcal{U}$, computable in $O(d)$ flops with no matrix
ever materialized. Stack $d/2$ of these on disjoint coordinate pairs with frequencies $\theta_i$ and
they commute, so

$$\mathbf{G}(n) = \prod_{i=1}^{d/2}\exp(n\theta_i\mathbf{L}_i)
  = \mathrm{blockdiag}\big(\mathbf{R}_2(n\theta_1),\dots,\mathbf{R}_2(n\theta_{d/2})\big).$$

That block-diagonal of $2\times2$ rotations *is* RoPE — the paper's Proposition 3.1 states RoPE is
**exactly** commuting multi-subspace GRAPE-M with the canonical coordinate pairs and a log-uniform
spectrum. What GRAPE adds is the freedom RoPE gives up: the planes and spectrum can be **learned**
(commuting subspaces at $O(d)$ per head), or you can allow a compact **non-commuting** mixture (at
$O(rd)$ per head) so different feature subspaces can *couple* — geometry the fixed RoPE basis cannot
express.

## Additive GRAPE: ALiBi and FoX are shears in a lifted space

To get an *additive* bias out of a *multiplicative* group, GRAPE uses the classic trick of a
**homogeneous lift**: augment $\mathbf{x}\in\mathbb{R}^d$ to $\hat{\mathbf{x}}\in\mathbb{R}^{d+k}$ and
work in $\mathrm{GL}(d+k)$ with a nilpotent generator. Because $\mathbf{A}^2 = \mathbf{0}$, the
exponential is just $\mathbf{G}_{\mathrm{add}}(n) = \mathbf{I} + n\omega\mathbf{A}$. With an asymmetric
lift $\hat{\mathbf{q}}_i = [\mathbf{q}_i;1;0]$, $\hat{\mathbf{k}}_j = [\mathbf{k}_j;0;1]$ and the rank-1
generator $\mathbf{A}_h = -\beta_h\,\mathbf{e}_{d+2}\mathbf{e}_{d+1}^\top$, the score becomes

$$\hat{\mathbf{q}}_i^\top\,\mathbf{G}_{\mathrm{add},h}(j-i)^{-\top}\,\hat{\mathbf{k}}_j
  = \mathbf{q}_i^\top\mathbf{k}_j + (j-i)\,\beta_h,$$

which is **exactly ALiBi** with head slope $\beta_h$. The nilpotent structure is not decoration: it is
what guarantees the exact relative law and clean streaming (cache the rotated keys once). GRAPE then
generalizes the *slope*. Replace the constant $\beta_h$ with non-negative softplus gates on the query
and key, and the bias becomes **content-dependent**:

$$\tilde{\mathbf{q}}_i^\top\tilde{\mathbf{k}}_j
  = \mathbf{q}_i^\top\mathbf{k}_j + (j-i)\,\omega\big[\mathrm{softplus}(\mathbf{v}^\top\mathbf{q}_i/\sqrt{d})
  + \mathrm{softplus}(\mathbf{u}^\top\mathbf{k}_j/\sqrt{d})\big].$$

This is **GRAPE-A-QK**: a learnable, content-adaptive linear bias derived from first principles rather
than hand-set per head. Drag the gate below to see the fixed ALiBi head-fan give way to a
content-driven slope:

<AdditiveBias />

The **Forgetting Transformer** falls out of the same picture. FoX's per-token forget gates accumulate
a bias $b_h(t,j) = \sum_{\ell=j+1}^{t}\log f_{\ell,h}$, which is precisely a *path product* of unipotent
factors $\prod_\ell(\mathbf{I} + \log f_{\ell,h}\,\mathbf{E}) = \mathbf{I} + b_h(t,j)\,\mathbf{E}$. So
FoX is an exact instance of **Path-Integral Additive GRAPE (GRAPE-AP)** — the endpoint-dependent
version that keeps row-wise composition and prefix-sum streaming.

## The whole map

Put the pieces together and every named scheme is a leaf on one tree. Click through them — each is
either *recovered exactly* by a specific generator or sits just past a known method as a GRAPE
*extension*:

<FamilyMap />

## Does the extra freedom help?

Here is where the honesty starts. GRAPE is validated at **small scale**: 353M and 770M models trained
on 50B tokens of FineWeb-Edu, context length 4,096, in a nanoGPT/Llama-style setup, evaluated 0-shot on
a standard NLU suite (ARC, HellaSwag, OBQA, PIQA, WinoGrande, SciQ). The training curves are close, but
GRAPE's additive variants hold a persistent small edge, and the authors note RoPE showed a training
instability at 770M that GRAPE did not:

<Figure
  src="/articles/grape-position-encoding/fig2.png"
  alt="Two line charts for the medium 353M model on FineWeb-Edu, training loss (left) and validation loss (right), versus training tokens from 0 to 50 billion. Four curves — RoPE (blue), ALiBi (green), FoX (orange), GRAPE-AP (red) — all decline from above 3.1 toward about 2.55–2.6 and stay tightly bunched, with GRAPE-AP and FoX slightly lower than RoPE late in training."
  caption="Training and validation loss for the 353M model across positional encodings; the curves are close, with GRAPE-AP tracking at or slightly below RoPE and ALiBi throughout (Zhang et al., 2026, Figure 2)."
/>

On downstream average, the ordering is consistent but the margins are small. For the 353M models,
GRAPE-AP (path-integral) is the best of the eight variants, edging FoX and ALiBi, with plain RoPE last:

<BenchBars
  title="353M models · average over 7 NLU tasks (0-shot, %)"
  unit="%"
  bars={[
    { label: "RoPE", value: 51.73 },
    { label: "ALiBi", value: 52.87 },
    { label: "FoX", value: 52.96 },
    { label: "GRAPE-AP", value: 53.25, highlight: true },
  ]}
/>

The 770M models tell the same story — GRAPE-AP first, RoPE last — again by roughly a point:

<BenchBars
  title="770M models · average over 7 NLU tasks (0-shot, %)"
  unit="%"
  bars={[
    { label: "RoPE", value: 55.76 },
    { label: "FoX", value: 56.30 },
    { label: "ALiBi", value: 56.44 },
    { label: "GRAPE-AP", value: 56.91, highlight: true },
  ]}
/>

<Callout type="warn">
**Read the wins narrowly.** (1) *Small scale, standard benchmarks.* Everything is 353M/770M on 50B
tokens at 4K context, on ARC/HellaSwag-style tasks — there are **no long-context or
length-extrapolation experiments** (no RULER, no retrieval), which is striking given the paper motivates
itself with long-context and ALiBi's extrapolation. (2) *The rotational story didn't pay off
empirically.* The Multiplicative variants that generalize RoPE — GRAPE-M-ctx/nonctx — actually
**underperform RoPE** on the large models (54.7–54.8 vs 55.76 avg); all the downstream gains come from
the **additive** family, so the framework's practical dominance rests on the ALiBi/FoX side, not the
rotation side. (3) *Margins are ~0.3–1.5 average points* over strong baselines, and the ranking flips
under the KV-shift setting: with KV-shift enabled, FoX edges GRAPE-AP at 770M (57.09 vs 56.86). (4)
*No efficiency measurements.* The $O(d)$/$O(rd)$-per-head costs are stated, not timed — there are no
wall-clock or FLOP comparisons. (5) Baselines are the authors' own reimplementations; there is no
comparison to tuned production models. The contribution is the **unifying theory and design space**,
lightly validated — not a demonstrated accuracy or efficiency SOTA.
</Callout>

## The take

GRAPE's real product is conceptual compression. Positional encoding stops being a list of tricks and
becomes a single knob — *which generator do you exponentiate?* — with RoPE, ALiBi and FoX as three
specific settings and a labelled space of alternatives (learned rotation bases, non-commuting mixtures,
content-gated slopes, path-integral biases) in between. That is genuinely clarifying, and the exact
recoveries are proved, not hand-waved: RoPE as commuting rank-2 rotations, ALiBi as a rank-1 unipotent
action, FoX as its path integral. What the paper does *not* yet show is that the new freedom the map
opens up buys much at scale — the strongest empirical variant is a modest improvement on the *additive*
side, the rotation-generalizing side trails plain RoPE, and the long-context claims the framing invites
go untested. As a theory it's a clean unification worth knowing; as a recipe, GRAPE-AP is a small,
honest win over FoX-style biases, and the rest is an invitation to experiment.

---

*Built on [Group Representational Position Encoding](https://arxiv.org/abs/2512.07805) (Zhang, Chen,
Liu, Qin, Yuan, Xu, Yuan, Gu, Yao; Princeton / UCLA / Tsinghua IIIS, ICLR 2026). Equations, tables and
figures are quoted from the paper (353M and 770M models, FineWeb-Edu, 0-shot lm-evaluation-harness);
the interactive diagrams are illustrations of the mechanism, not measured data. Related reading:
[how attention works](/articles/how-transformers-attention-works),
[a tour of attention mechanisms](/articles/attention-mechanisms),
[MiniMax Sparse Attention](/articles/minimax-sparse-attention), and
[how LLM inference works](/articles/how-llm-inference-works).*