Tapered Language Models: spend your width where the work is

2026-07-01 · 8 min · llm · transformers · architecture · scaling · explainer

Here's an assumption baked into every transformer, recurrent, and memory-based language model since 2017: the layers are identical. Same residual dimension, same attention shape, same MLP width, stacked N deep. It's a default inherited from the original transformer and almost never questioned — parameters are spread uniformly across depth.

Tapered Language Models (TLMs), from Bayat, Behrouz, and Courville, ask what happens if you don't. Their answer is a one-line change with a free-lunch flavor: under a fixed parameter budget, give the early layers more MLP width and the late layers less, on a smooth cosine schedule. Same parameters, same FLOPs — better model.

MLP width across depth · same total budget, redistributed

taper strength3.0:1 early:late

total MLP params 1.00× — held fixedearly : late width 3.0 : 1

Tapered: a cosine schedule pours width into the early layers and thins the late ones. The total stays exactly the same — this is a redistribution of a fixed budget, not extra parameters, so FLOPs and param count don't change.

The default no one questions

Why would non-uniform be better? Because a growing pile of evidence says layers don't contribute equally. The paper points at three findings:

Early-exit methods show the residual stream often converges to its final prediction well before the last layer.
Layer-skipping shows later layers can be bypassed at inference with minimal damage.
Interpretability work finds lower layers capture shallow syntactic patterns while upper layers encode semantics — different jobs, not equal ones.

The unifying picture: later layers refine the residual stream rather than transform it. An early layer rewrites the representation; a late layer nudges it. If you measure how much each layer actually changes things, the work is front-loaded:

where the work happens · representation change vs allocated width, by depth

layer (drag to inspect)L2

Later layers largely refine rather than transform — they nudge the residual stream where early layers rewrite it. Uniform allocation ignores that and spends equal capacity everywhere. Tapering pours width into the layers doing the heavy lifting, which is why the same parameter budget buys a better model.

And if late layers are only refining, giving them the same width as the layers doing real transformation is wasted capacity — capacity you could have spent where it matters.

The taper, precisely

TLMs taper exactly one thing: the MLP intermediate width $d_{ff}$ . Attention — residual dimension, head count, key/value dims — is left identical to the baseline. That's a deliberate choice: MLPs dominate the parameter count in every modern LM family, and width is a single clean axis to vary. The schedule is a cosine over depth:

d_{ff}(l) = d_{end} + \frac{d_{start} - d_{end}}{2}\left(1 + \cos\frac{\pi l}{L-1}\right)

Layer 0 is widest ( $d_{ff} = d_{start}$ ), the last layer narrowest ( $d_{ff} = d_{end}$ ), monotonically decreasing in between. The endpoints are multipliers of the baseline width, and the best config is $1.5\times$ at the front, $0.5\times$ at the back — a 3:1 taper. The crucial property: the per-layer widths average to the baseline, so

\frac{1}{L}\sum_l d_{ff}(l) = d_{ff}^{\text{baseline}}

Early layers get wider than baseline, late layers narrower, and the integral is preserved. Total parameters don't change. Total FLOPs don't change. Tapering only shifts where the compute is spent, not how much. That's what makes it a redistribution rather than a bigger model — and why the comparison to the uniform baseline is honest.

Left: per-layer MLP intermediate width for a uniform baseline and three tapered schedules (step-wise, linear, cosine) on a 440M Transformer, drawn as stacked bar widths that shrink toward the top for the tapered variants. Right: validation perplexity versus taper strength, with cosine dropping from the 16.28 uniform baseline to a 14.44 minimum at the 1.5→0.5 range while linear stays near baseline. — All schedules share the same total parameter count and FLOPs; the cosine taper reaches the lowest perplexity, 14.44 vs the 16.28 uniform baseline (paper, Figure 1).

Direction is everything

The foundational experiment splits the stack into three blocks and moves the extra capacity around — early, middle, or late — at fixed budget. The result is unambiguous:

where should the capacity go? · 440M, fixed budget

best: −0.32 ppl vs uniform, same params

Same parameter budget every time — only the placement changes. Front-loading capacity helps; centering it is worse than uniform; and back-loading (wider-late) is the worst of all, +1.01 perplexity over doing nothing. Where you spend the width matters, and the direction is unambiguous.

Front-loading helps. Centering the capacity is worse than uniform. And back-loading — wider late layers — is the worst option of all, +1.01 perplexity over just doing nothing. This is the control that makes the whole paper: the gain isn't from "more capacity somewhere," it's specifically from putting it where the representational work happens.

A 440M Transformer's layers split into three equal blocks (early, middle, late), each block's MLP width scaled while total parameters stay fixed. Wider-early gives 15.96 perplexity (best, green), wider-late 17.29 (worst), wider-middle 16.61, all compared against the 16.28 uniform baseline. — Moving the same extra MLP capacity to the early layers helps (15.96), while back-loading it hurts most (17.29) — the control that isolates direction from raw capacity (paper, Figure 2).

The shape matters too. Sweeping cosine against linear and sigmoid schedules, cosine wins in every setting; sigmoid is often worse than the uniform baseline. And taper strength is U-shaped — too gentle leaves gains on the table, too aggressive (a 7:1 ratio) starves the late layers and regresses. The 1.5→0.5 cosine is the bottom of that U.

How consistently it holds

The tuned config — cosine 1.5→0.5, found once on a 440M Transformer — is then transferred unchanged to three scales (440M/30B tokens, 760M/50B, 1.3B/100B) and four architectures: plain Transformer, Gated Attention, and Behrouz's own HOPE and Titans memory models. It keeps improving almost everywhere. At 760M, the perplexity reduction from the exact same parameter budget:

WikiText perplexity reduction from tapering — 760M (higher = bigger drop)

Titans

0.81

Gated Attention

0.76

Transformer

0.44

Hope-attention

0.12

00.51

And it carries through to downstream accuracy — the average over eight commonsense benchmarks (LAMBADA, PIQA, HellaSwag, WinoGrande, ARC-easy/challenge, SIQA, BoolQ) rises for every architecture at 760M:

Commonsense accuracy gain from tapering — 760M (percentage points)

Titans

0.99pp

Transformer

0.59pp

Hope-attention

0.36pp

Gated Attention

0.27pp

00.51

The full picture, uniform → tapered, is small-but-consistent rather than dramatic:

scale	architecture	WikiText ppl	LAMBADA ppl	commonsense avg
760M	Transformer	21.86 → 21.42	22.29 → 21.25	52.25 → 52.84
760M	Gated Attention	20.74 → 19.98	21.85 → 21.44	52.61 → 52.88
760M	Titans	21.58 → 20.77	23.09 → 22.92	52.30 → 53.29
1.3B	Transformer	17.39 → 17.17	17.62 → 16.93	56.05 → 56.38
1.3B	Titans	16.05 → 15.76	14.19 → 14.04	56.73 → 57.08

The honest read: at scale the gains are typically 0.1–1.0 perplexity and a few tenths of a point of accuracy — improving in ~15 of 16 measured cells (the lone regression is 1.3B HOPE's WikiText, off by 0.03). It's not a step change. It's a free, universal nudge in the right direction from a config that was never even tuned at these scales.

Why it works

The mechanism check makes the story tight. Measure the cosine similarity between each MLP's output and the residual stream it writes into, and it rises with depth — later MLPs produce updates increasingly aligned with the residual (Pearson r ≈ 0.49–0.71 vs. layer index). An update aligned with the residual is a refinement; an orthogonal one is a transformation. So late MLPs, writing residual-aligned updates, aren't using their extra width — the hidden dimension is spent nudging in a direction the stream already points. Tapering removes width exactly where that alignment is highest and moves it to the early layers, which write the orthogonal, representation-defining updates that actually need the capacity. The MLP is where the parameters live; this is just allocating them by how hard each layer is working.

Two line plots of cosine similarity versus relative layer depth across the GPT-2 family (124M to 1.5B). Left: block updates versus the residual stream. Right: MLP output versus the residual stream. Both curves rise toward 1.0 at greater depth, showing later-layer updates increasingly align with the residual they write into. — Later layers' updates grow more aligned (higher cosine similarity) with the residual stream they write into, marking them as refinements rather than transformations (paper, Figure 4).

The take

I like this paper for the same reason I liked HydraHead: the architectural change is derived from a measurement, not reverse-justified. "Later layers refine, not transform" is an old observation; TLMs turn it into a concrete lever — cosine-taper the MLP width — and then run the control (reverse it, and it hurts) that proves the direction is what's doing the work.

The caveats are real and the authors state them. The gains at scale are modest, the single 1.5→0.5 cosine schedule was tuned only on a 440M Transformer and transferred without re-tuning (so there's likely more on the table), and there's no code release. But the appeal is that it costs nothing — same parameters, same FLOPs, one function applied to the MLP widths. For a default that's gone unquestioned for eight years, "uniform depth was leaving free perplexity on the floor" is a satisfying result, and an easy one to try.

Built on Tapered Language Models (Reza Bayat, Ali Behrouz, Aaron Courville, 2026). Perplexity, accuracy, and ablation figures are quoted from the paper's tables; the residual-alignment correlation is from its Figure 4.