# Tapered Language Models: spend your width where the work is

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/tapered-language-models
> date: 2026-07-01
> tags: llm, transformers, architecture, scaling, explainer
Here's an assumption baked into every transformer, recurrent, and memory-based language
model since 2017: the layers are **identical**. Same residual dimension, same attention
shape, same MLP width, stacked N deep. It's a default inherited from the original
transformer and almost never questioned — parameters are spread *uniformly* across depth.

**Tapered Language Models** (TLMs), from Bayat, Behrouz, and Courville, ask what happens if
you don't. Their answer is a one-line change with a free-lunch flavor: under a *fixed*
parameter budget, give the early layers more MLP width and the late layers less, on a smooth
cosine schedule. Same parameters, same FLOPs — better model.

<TaperSchedule />

## The default no one questions

Why would non-uniform be better? Because a growing pile of evidence says layers *don't*
contribute equally. The paper points at three findings:

- **Early-exit** methods show the residual stream often converges to its final prediction
  well before the last layer.
- **Layer-skipping** shows later layers can be bypassed at inference with minimal damage.
- Interpretability work finds lower layers capture shallow syntactic patterns while upper
  layers encode semantics — different jobs, not equal ones.

The unifying picture: **later layers refine the residual stream rather than transform it.**
An early layer rewrites the representation; a late layer nudges it. If you measure how much
each layer actually changes things, the work is front-loaded:

<WhyEarly />

And if late layers are only refining, giving them the *same* width as the layers doing real
transformation is wasted capacity — capacity you could have spent where it matters.

## The taper, precisely

TLMs taper exactly one thing: the **MLP intermediate width** $d_{ff}$. Attention — residual
dimension, head count, key/value dims — is left identical to the baseline. That's a
deliberate choice: MLPs dominate the parameter count in every modern LM family, and width is
a single clean axis to vary. The schedule is a cosine over depth:

$$
d_{ff}(l) = d_{end} + \frac{d_{start} - d_{end}}{2}\left(1 + \cos\frac{\pi l}{L-1}\right)
$$

Layer 0 is widest ($d_{ff} = d_{start}$), the last layer narrowest ($d_{ff} = d_{end}$),
monotonically decreasing in between. The endpoints are multipliers of the baseline width, and
the best config is **$1.5\times$ at the front, $0.5\times$ at the back** — a 3:1 taper. The
crucial property: the per-layer widths **average to the baseline**, so

$$
\frac{1}{L}\sum_l d_{ff}(l) = d_{ff}^{\text{baseline}}
$$

Early layers get wider than baseline, late layers narrower, and the integral is preserved.
Total parameters don't change. Total FLOPs don't change. Tapering only shifts *where* the
compute is spent, not how much. That's what makes it a redistribution rather than a bigger
model — and why the comparison to the uniform baseline is honest.

<Figure
  src="/articles/tapered-language-models/fig1.png"
  alt="Left: per-layer MLP intermediate width for a uniform baseline and three tapered schedules (step-wise, linear, cosine) on a 440M Transformer, drawn as stacked bar widths that shrink toward the top for the tapered variants. Right: validation perplexity versus taper strength, with cosine dropping from the 16.28 uniform baseline to a 14.44 minimum at the 1.5→0.5 range while linear stays near baseline."
  caption="All schedules share the same total parameter count and FLOPs; the cosine taper reaches the lowest perplexity, 14.44 vs the 16.28 uniform baseline (paper, Figure 1)."
/>

## Direction is everything

The foundational experiment splits the stack into three blocks and moves the extra capacity
around — early, middle, or late — at fixed budget. The result is unambiguous:

<TaperDirection />

Front-loading helps. Centering the capacity is *worse* than uniform. And back-loading —
wider late layers — is the worst option of all, **+1.01 perplexity** over just doing nothing.
This is the control that makes the whole paper: the gain isn't from "more capacity somewhere,"
it's specifically from putting it where the representational work happens.

<Figure
  src="/articles/tapered-language-models/fig2.png"
  alt="A 440M Transformer's layers split into three equal blocks (early, middle, late), each block's MLP width scaled while total parameters stay fixed. Wider-early gives 15.96 perplexity (best, green), wider-late 17.29 (worst), wider-middle 16.61, all compared against the 16.28 uniform baseline."
  caption="Moving the same extra MLP capacity to the early layers helps (15.96), while back-loading it hurts most (17.29) — the control that isolates direction from raw capacity (paper, Figure 2)."
/>

The *shape* matters too. Sweeping cosine against linear and sigmoid schedules, cosine wins in
every setting; sigmoid is often worse than the uniform baseline. And taper strength is
U-shaped — too gentle leaves gains on the table, too aggressive (a 7:1 ratio) starves the late
layers and regresses. The 1.5→0.5 cosine is the bottom of that U.

## How consistently it holds

The tuned config — cosine 1.5→0.5, found once on a 440M Transformer — is then transferred
*unchanged* to three scales (440M/30B tokens, 760M/50B, 1.3B/100B) and four architectures:
plain Transformer, Gated Attention, and Behrouz's own **HOPE** and **Titans** memory models.
It keeps improving almost everywhere. At 760M, the perplexity reduction from the exact same
parameter budget:

<BenchBars
  title="WikiText perplexity reduction from tapering — 760M (higher = bigger drop)"
  unit=""
  bars={[
    { label: "Titans", value: 0.81, highlight: true },
    { label: "Gated Attention", value: 0.76, highlight: true },
    { label: "Transformer", value: 0.44, highlight: true },
    { label: "Hope-attention", value: 0.12, highlight: true },
  ]}
/>

And it carries through to downstream accuracy — the average over eight commonsense benchmarks
(LAMBADA, PIQA, HellaSwag, WinoGrande, ARC-easy/challenge, SIQA, BoolQ) rises for every
architecture at 760M:

<BenchBars
  title="Commonsense accuracy gain from tapering — 760M (percentage points)"
  unit="pp"
  bars={[
    { label: "Titans", value: 0.99, highlight: true },
    { label: "Transformer", value: 0.59, highlight: true },
    { label: "Hope-attention", value: 0.36, highlight: true },
    { label: "Gated Attention", value: 0.27, highlight: true },
  ]}
/>

The full picture, uniform → tapered, is small-but-consistent rather than dramatic:

| scale | architecture | WikiText ppl | LAMBADA ppl | commonsense avg |
|---|---|---|---|---|
| 760M | Transformer | 21.86 → **21.42** | 22.29 → **21.25** | 52.25 → **52.84** |
| 760M | Gated Attention | 20.74 → **19.98** | 21.85 → **21.44** | 52.61 → **52.88** |
| 760M | Titans | 21.58 → **20.77** | 23.09 → **22.92** | 52.30 → **53.29** |
| 1.3B | Transformer | 17.39 → **17.17** | 17.62 → **16.93** | 56.05 → **56.38** |
| 1.3B | Titans | 16.05 → **15.76** | 14.19 → **14.04** | 56.73 → **57.08** |

The honest read: at scale the gains are typically 0.1–1.0 perplexity and a few tenths of a
point of accuracy — improving in ~15 of 16 measured cells (the lone regression is 1.3B
HOPE's WikiText, off by 0.03). It's not a step change. It's a **free, universal nudge in the
right direction** from a config that was never even tuned at these scales.

## Why it works

The mechanism check makes the story tight. Measure the cosine similarity between each MLP's
output and the residual stream it writes into, and it *rises* with depth — later MLPs produce
updates increasingly *aligned* with the residual (Pearson r ≈ 0.49–0.71 vs. layer index). An
update aligned with the residual is a refinement; an orthogonal one is a transformation. So
late MLPs, writing residual-aligned updates, aren't using their extra width — the hidden
dimension is spent nudging in a direction the stream already points. Tapering removes width
exactly where that alignment is highest and moves it to the early layers, which write the
orthogonal, representation-defining updates that actually need the capacity. The
[MLP is where the parameters live](/articles/mixture-of-experts-from-scratch); this is just
allocating them by how hard each layer is working.

<Figure
  src="/articles/tapered-language-models/fig3.png"
  alt="Two line plots of cosine similarity versus relative layer depth across the GPT-2 family (124M to 1.5B). Left: block updates versus the residual stream. Right: MLP output versus the residual stream. Both curves rise toward 1.0 at greater depth, showing later-layer updates increasingly align with the residual they write into."
  caption="Later layers' updates grow more aligned (higher cosine similarity) with the residual stream they write into, marking them as refinements rather than transformations (paper, Figure 4)."
/>

## The take

I like this paper for the same reason I liked [HydraHead](/articles/hydrahead): the
architectural change is *derived from* a measurement, not reverse-justified. "Later layers
refine, not transform" is an old observation; TLMs turn it into a concrete lever — cosine-taper
the MLP width — and then run the control (reverse it, and it hurts) that proves the direction
is what's doing the work.

The caveats are real and the authors state them. The gains at scale are modest, the single
1.5→0.5 cosine schedule was tuned only on a 440M Transformer and transferred without
re-tuning (so there's likely more on the table), and there's no code release. But the appeal
is that it costs *nothing* — same parameters, same FLOPs, one function applied to the MLP
widths. For a default that's gone unquestioned for eight years, "uniform depth was leaving
free perplexity on the floor" is a satisfying result, and an easy one to try.

---

*Built on [Tapered Language Models](https://arxiv.org/abs/2606.23670) (Reza Bayat, Ali
Behrouz, Aaron Courville, 2026). Perplexity, accuracy, and ablation figures are quoted from
the paper's tables; the residual-alignment correlation is from its Figure 4.*