DCFormer: an ICML oral that let attention heads borrow each other's circuits, and quietly shipped anyway

2026-08-03 · 13 min · attention · transformers · architecture · scaling · llm

Dynamically Composable Multi-Head Attention (DCMHA) is a two-year-old idea: May 2024, an ICML oral, a real mechanism, a real headline number — DCPythia-6.9B beats open Pythia-12B on Pile validation perplexity (5.95 vs 6.01) with close to half the parameters. By any normal measure that should have traveled. Semantic Scholar currently shows it at roughly a dozen citations, one flagged "influential." That's a modest footprint for an ICML oral two years out. What it does have is a real deployment: Caiyun Technology, the authors' own industry affiliation, put it into a production language model and an AI-RPG platform, and kept extending the training code through early 2025. The honest frame is not "field-changing" — it's a technically solid piece of work that quietly shipped without the citation graph to match, and is worth understanding on its own terms.

Two problems with heads that never talk to each other

Standard multi-head attention runs $h$ heads in parallel, each on its own learned $Q$ / $K$ / $V$ projection, and concatenates the results. The heads never see each other's work. That independence is also the design's two weaknesses. First, the low-rank bottleneck: each head's attention score matrix is a rank-limited function of a head-dimension-sized projection, and Bhojanapalli et al. (2020) showed that widening the per-head QK dimension relieves it — but widening every head's projection is expensive. Second, head redundancy: with nothing coupling them, heads are free to learn overlapping, partially duplicated functions instead of covering the space of useful attention patterns efficiently.

DCMHA's answer isn't to make heads bigger. It's to let them compose — combine their scores and weights across heads, per token — which the paper shows buys the same kind of expressivity gain that a wider QK projection would, without actually widening anything.

What Compose does

Two calls to a function named Compose are inserted into ordinary multi-head attention: one right after the scores are computed (pre-softmax), one right after softmax turns them into weights (post-softmax).

Two-panel diagram. Left, (a): the overall DCMHA architecture — Q, K, V are projected and split per head, batched matrix multiplication produces per-head attention scores, then two Compose blocks (one before softmax, one after) sit between the score computation and the final merge-and-project step, labeled as channel-mixing and head-mixing operations. Right, (b): the internals of Compose — an attention vector A with one entry per head, drawn as a strip of gray squares, feeds into five branches labeled B1 through B5 (base projection, key-wise dynamic gating, key-wise dynamic projection, query-wise dynamic projection, query-wise dynamic gating), each producing a small green vector of dynamic weights generated from the query or key vectors, all summing into a new attention vector A-prime. — (a) Two Compose calls sit inside ordinary multi-head attention, one on the scores and one on the weights. (b) Inside Compose: five branches — a static base plus four token-conditioned ones — recombine one head's attention vector using every other head's (Xiao et al., 2024, Figure 2).

For a fixed query/key pair, stack the $H$ heads' scores (or weights) into one vector $A_{:ij} \in \mathbb{R}^H$ — "the attention vector." Compose turns that into a new vector $A'_$ by summing five branches:

B1, a static base projection (in practice, DCMHA drops this in favor of a plain skip connection, with no measurable loss);
B2/B3, a query-wise dynamic low-rank projection and a query-wise dynamic gate, both generated from $Q_i$ by a small FFN;
B4/B5, the same pair generated from $K_j$ instead.

Every one of B2 through B5 is input-dependent — the weights that decide how much of head $h'$ leaks into head $h$ 's new value are computed fresh from the actual query or key vector at that position, not fixed at training time.

compose: one head reading another head’s circuit, per token

borrow strength α = 0.52

← h1, α=0.52

← h2, α=0.52

← h3, α=0.52

← h4, α=0.52

← h0, α=0.52

pre-Compose post-Compose

Fix one physical circuit — head h reads head h+1’s score for this query/key pair — and only change the token. At “cat” the borrowing strength is α = 0.52, generated from the query/key vectors by a small FFN inside Compose, not looked up from a fixed table. Click through the tokens: the pre-Compose bars (top, gray) never move — they are what the ordinary Q/K/V projections already computed — but the post-Compose bars (bottom, teal) redistribute differently at every token, because α does. That per-token swing is the entire difference between DCMHA and Talking-Heads Attention, which mixes heads with the same fixed matrix for every token, every input, forever.

Why it has to be dynamic, not just wider

The paper proves something specific about the static version of this idea first. Compose one head's score with a fixed matrix $C \in \mathbb{R}^{H \times H}$ , and that is provably identical to concatenating an $H$ -fold expanded QK projection (Theorem 2.1); do the same to the post-softmax weights, and it's identical to an expanded V/O projection (Theorem 2.2). In other words: a static composition matrix buys you exactly what a wider head dimension buys you — the fix for the low-rank bottleneck — and nothing more.

Talking-Heads Attention (Shazeer et al., 2020) is that static case: it already composes both scores and weights, just with one fixed matrix reused for every token, every input, forever. DCMHA's own ablation measures the gap this leaves on the table — adding the static projection alone gets Pile validation perplexity from 11.68 down to 11.17, but the full dynamic Compose reaches 10.79. The static version is doing real work; the dynamic version is doing about 60% more of it, by this measure. Query-wise and key-wise branches contribute nearly as well on their own as together, and post-compose (on the weights) alone beats pre-compose (on the scores) alone, 11.05 vs 11.54 — the low- rank projection branches (B2/B4) matter more than the gates (B3/B5).

Why it's cheap

The reason DCMHA doesn't cost what a full head-to-head transform would is a decomposition, not a shortcut. Conceptually, composing every head with every other head needs an $H \times H$ transform per query/key pair — quadratic in the number of heads. DCMHA factors that tensor into a query-wise term plus a key-wise term (row + column), and factors each of those into a rank- $R$ product plus a diagonal gate (low-rank + diagonal decomposition). The cost drops from $H^2$ to $2HR + H$ , and — a nice side effect — the key-wise half can be computed once and cached alongside K/V, which is exactly what a serving stack needs.

cost of composing heads: full tensor vs. low-rank + diagonal

heads (H)8

rank (R)

naive H² tensor

DCMHA 2HR+H

DCMHA cost / naive cost = 62.5%extra params at D_h=128: 1.30%

The naive way to let every head read every other head is a transform tensor of size H², applied at every query/key pair — quadratic in head count. DCMHA never forms it: the query-wise and key-wise Compose terms are each a rank-2 product plus a diagonal gate, so the real cost is 2HR + H. At H = 8, R = 2, that is 40 instead of 64 — and the gap only grows as H scales up, which is why DCMHA stays cheap at 32 and 64 heads where a full H×H tensor would not. The paper reports this concretely at 6.9B scale (D_h = 128, R = 2): about 1.3% extra parameters and 1.9–3.3% extra FLOPs, depending on sequence length.

At the paper's own 6.9B-scale example ( $D_h = 128$ , $R = 2$ ): roughly 1.3% extra parameters and 1.9–3.3% extra FLOPs, depending on sequence length. Rank $R = 2$ turns out to be close to a sweet spot in the ablation ( $R{=}1$ : 10.87 ppl, $R{=}2$ : 10.83, $R{=}4$ : 10.89 — non-monotonic, and not worth pushing higher).

The headline number

Trained on The Pile, matched Chinchilla-style token budgets, three model families: the scaling curves show DCFormer-834M matches a plain Transformer trained with roughly 1.87 times the compute, and DCFormer++ (RoPE + SwiGLU added to both sides) matches its own baseline at roughly 1.67 times. That gap doesn't shrink with scale — DCMHA's relative improvement decays more slowly than the RoPE+SwiGLU improvement does, which is the favorable direction.

The result that carries the abstract is the 300B-token run against the actual Pythia suite:

Line chart with compute on the x-axis (2.8B times 300B, 6.9B times 300B, 12B times 300B tokens) and loss on the y-axis. Teal circles mark actual Pythia models at each compute point, connected by a dashed fitted trend line. Orange stars mark DCPythia at the 2.8B and 6.9B points, each notably lower than the Pythia line at the same compute. Two horizontal arrows connect each DCPythia point across to where the Pythia trend line reaches the same loss, labeled 1.85x and 1.97x respectively. — DCPythia reaches a given loss at less compute than Pythia needs for the same loss — and the multiplier grows from 1.85× at the 2.8B scale to 1.97× at 6.9B, not shrinks (Xiao et al., 2024, Figure 4).

Pile validation perplexity, 6.9B-class models

Pythia-6.9B

6.29

DCPythia-6.9B

5.95

Pythia-12B

6.01

02468

DCPythia-6.9B's 5.95 beats Pythia-12B's 6.01 — a model with close to half the parameters, ahead on the metric that matters for pretraining. It also edges out on average 0-shot downstream accuracy (56.7 vs 56.5) and 5-shot (57.7 vs 57.2). The gap over its own size class is not close: Pythia-6.9B sits at 6.29, meaningfully behind.

Model	Pile ppl	Flan ppl	Avg 0-shot acc
Pythia-2.8B	6.63	8.16	53.1
DCPythia-2.8B	6.36	7.68	54.5
Pythia-6.9B	6.29	7.85	55.1
DCPythia-6.9B	5.95	7.13	56.7
Pythia-12B	6.01	—	56.5

The gap is largest on the Flan Collection (instruction-following/few-shot/CoT data) and grows with scale, which the authors read as DCMHA disproportionately helping the harder, more compositional end of the task distribution — a reading the paper backs up with a purpose-built test.

A synthetic test built to need composition

The authors built a 74-task, 888-example diagnostic where getting the right answer requires simultaneously attending to the right source token and applying the right output transformation (e.g., mapping an object to its superclass) — precisely the combination a head that only ever reads its own fixed QK/OV circuit should struggle with:

Synthetic composition task — accuracy

Pythia-6.9B

31.9%

DCPythia-6.9B

39%

010203040

Perplexity on this set drops from 10.05 to 7.36 alongside the accuracy jump — a much bigger swing than on Pile or Flan, and the paper's own explanation is the one you'd expect: this task rewards recombining an existing head's QK circuit with a different head's OV circuit on the fly, which is the one thing static heads structurally cannot do. The head-diversity analysis (captured variance of concatenated QK and OV projection matrices, lower meaning more diverse heads) backs this qualitatively too — DCPythia shows markedly more QK-circuit diversity than Pythia, and moderately more OV-circuit diversity.

The honest costs

None of this is free, and the paper says so plainly. Composition is I/O-bound, not compute-bound, and the reference implementation has no fused kernel — plain JAX for training, plain PyTorch for inference:

Size	Training throughput (DCFM++ / TFM++)	Inference throughput (DCFM++ / TFM++)
2.8B	74.5%	81–88%
6.9B	83.1%	89–95%
13B	84.4%	90–95%
33B	89.2%	90–95%

The overhead shrinks as models scale up, and the authors are explicit that a fused kernel — FlashAttention-style — is headroom they haven't taken. A separate lever recovers most of it directly: raising the local-to-global attention ratio and composing only query-wise (dropping the key-wise branches) pushes DCFormer++-6.9B's training throughput from 83.1% back up to 92.5% of baseline, at a small, still-net-favorable cost in perplexity.

The mechanism transfers outside language too: on ImageNet-1K, DCViT-S/16 at 1.03× the baseline's parameters (68.0 top-1 at epoch 90) matches ViT-M/16 at 1.72× the parameters (67.1 top-1) — the same roughly 1.7× parameter-efficiency story, in a different domain, on one held-out test.

A different axis from the memory-side attention variants

If you've read the field guide to attention mechanisms on this site, it's worth being precise about where DCMHA sits relative to that map. MQA, GQA, and MLA all operate on what that piece calls the memory axis — they share or compress the K/V heads to shrink the KV cache, trading some quality for less memory bandwidth at decode time. DCMHA doesn't touch the cache at all: the number of physical K/V heads is unchanged, nothing shrinks. It operates on an orthogonal axis entirely — not how many heads you cache, but what each head is allowed to compute, by letting it borrow another head's QK or OV circuit, per token. A model could in principle combine GQA's cache savings with DCMHA's composition; the paper doesn't test that combination, so read it as plausible, not demonstrated. (For the general design question of specializing attention below the layer level, HydraHead is the other piece on this site working that seam, from a different angle.) The broader landscape of architecture choices — attention, position encoding, MoE, diffusion — is mapped at /architectures.

How much of this actually caught on

Two years after an ICML oral, roughly a dozen citations and one flagged "influential" is a modest academic footprint — the kind of number that would normally suggest an idea that didn't pan out. What actually happened looks different: Caiyun Technology, the paper's own industry co-author's employer, shipped DCFormer into a production language model and upgraded an AI-RPG platform to run on it, and the GitHub repository was still being extended — DeepSpeed ZeRO support, Hugging Face Trainer integration — well into 2025. No successor paper has benchmarked against DCFormer as a state-of-the-art baseline to beat; the adoption signal here is industrial, not academic. That's a real but different kind of validation than a citation count measures, and it's the honest way to read this one: not a paper that changed the field's direction, but a working piece of architecture that one production system actually adopted, sitting quietly under-cited.

The take

Fixed, independent attention heads leave two things on the table: a low-rank bottleneck that a wider head dimension fixes at a real cost, and redundancy nothing forces heads to avoid. DCMHA's Compose function fixes both by recombining heads' scores and weights per token, through a decomposition cheap enough that a 6.9B model pays about 1.3% more parameters for it. The result — DCPythia-6.9B beating Pythia-12B on perplexity at roughly half the parameters — is real, reproduced by the authors under controlled settings, and backed by a synthetic test built specifically to need what static heads can't do. It just hasn't been the paper everyone cites. Production adoption at one company and a quiet GitHub repository are what two years actually bought it — which is a fine outcome for a piece of architecture, even if it isn't the one the citation count would lead you to expect.

Built on Improving Transformers with Dynamically Composable Multi-Head Attention (Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan; Beijing University of Posts and Telecommunications / Caiyun AI; ICML 2024, Oral) and its code release. Figures are the paper's own Figures 2 and 4, reproduced for commentary. Tables and numbers are the authors' except where marked as this site's own illustrative simplification (the Compose bar demo, the head-cost demo); interactive diagrams are mine.