# DCFormer: an ICML oral that let attention heads borrow each other's circuits, and quietly shipped anyway

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/dcformer
> date: 2026-08-03
> tags: attention, transformers, architecture, scaling, llm
**Dynamically Composable Multi-Head Attention** (DCMHA) is a two-year-old idea: May 2024,
an ICML oral, a real mechanism, a real headline number — DCPythia-6.9B beats open
Pythia-12B on Pile validation perplexity (5.95 vs 6.01) with close to half the
parameters. By any normal measure that should have traveled. Semantic Scholar currently
shows it at roughly a dozen citations, one flagged "influential." That's a modest
footprint for an ICML oral two years out. What it does have is a real deployment: Caiyun
Technology, the authors' own industry affiliation, put it into a production language
model and an AI-RPG platform, and kept extending the training code through early 2025.
The honest frame is not "field-changing" — it's a technically solid piece of work that
quietly shipped without the citation graph to match, and is worth understanding on its
own terms.

<Callout type="warn">
There is a **second, unrelated** paper that reuses the same name: "DCFormer: Efficient 3D
Vision-Language Modeling with Decomposed Convolutions" (arXiv 2502.05091, 2025) is about
decomposed convolutions for 3D vision-language models — nothing to do with attention
heads. Everything below is arXiv 2405.08553, Xiao, Meng, Li, and Yuan, "Improving
Transformers with Dynamically Composable Multi-Head Attention." Check the ID if you go
looking for it.
</Callout>

## Two problems with heads that never talk to each other

Standard multi-head attention runs $h$ heads in parallel, each on its own learned
$Q$/$K$/$V$ projection, and concatenates the results. The heads never see each other's
work. That independence is also the design's two weaknesses. First, the **low-rank
bottleneck**: each head's attention score matrix is a rank-limited function of a
head-dimension-sized projection, and Bhojanapalli et al. (2020) showed that widening the
per-head QK dimension relieves it — but widening every head's projection is expensive.
Second, **head redundancy**: with nothing coupling them, heads are free to learn
overlapping, partially duplicated functions instead of covering the space of useful
attention patterns efficiently.

DCMHA's answer isn't to make heads bigger. It's to let them **compose** — combine their
scores and weights across heads, per token — which the paper shows buys the same kind of
expressivity gain that a wider QK projection would, without actually widening anything.

## What Compose does

Two calls to a function named `Compose` are inserted into ordinary multi-head attention:
one right after the scores are computed (pre-softmax), one right after softmax turns
them into weights (post-softmax).

<Figure
  src="/articles/dcformer/fig1.png"
  alt="Two-panel diagram. Left, (a): the overall DCMHA architecture — Q, K, V are projected and split per head, batched matrix multiplication produces per-head attention scores, then two Compose blocks (one before softmax, one after) sit between the score computation and the final merge-and-project step, labeled as channel-mixing and head-mixing operations. Right, (b): the internals of Compose — an attention vector A with one entry per head, drawn as a strip of gray squares, feeds into five branches labeled B1 through B5 (base projection, key-wise dynamic gating, key-wise dynamic projection, query-wise dynamic projection, query-wise dynamic gating), each producing a small green vector of dynamic weights generated from the query or key vectors, all summing into a new attention vector A-prime."
  caption="(a) Two Compose calls sit inside ordinary multi-head attention, one on the scores and one on the weights. (b) Inside Compose: five branches — a static base plus four token-conditioned ones — recombine one head's attention vector using every other head's (Xiao et al., 2024, Figure 2)."
/>

For a fixed query/key pair, stack the $H$ heads' scores (or weights) into one vector
$A_{:ij} \in \mathbb{R}^H$ — "the attention vector." `Compose` turns that into a new
vector $A'_{:ij}$ by summing five branches:

- **B1**, a static base projection (in practice, DCMHA drops this in favor of a plain
  skip connection, with no measurable loss);
- **B2/B3**, a query-wise dynamic low-rank projection and a query-wise dynamic gate,
  both generated from $Q_i$ by a small FFN;
- **B4/B5**, the same pair generated from $K_j$ instead.

Every one of B2 through B5 is **input-dependent** — the weights that decide how much of
head $h'$ leaks into head $h$'s new value are computed fresh from the actual query or key
vector at that position, not fixed at training time.

<ComposeVectors />

## Why it has to be dynamic, not just wider

The paper proves something specific about the *static* version of this idea first.
Compose one head's score with a fixed matrix $C \in \mathbb{R}^{H \times H}$, and that is
provably identical to concatenating an $H$-fold expanded QK projection (Theorem 2.1); do
the same to the post-softmax weights, and it's identical to an expanded V/O projection
(Theorem 2.2). In other words: a **static** composition matrix buys you exactly what a
wider head dimension buys you — the fix for the low-rank bottleneck — and nothing more.

**Talking-Heads Attention** (Shazeer et al., 2020) is that static case: it already
composes both scores and weights, just with one fixed matrix reused for every token,
every input, forever. DCMHA's own ablation measures the gap this leaves on the table —
adding the static projection alone gets Pile validation perplexity from 11.68 down to
11.17, but the full dynamic Compose reaches 10.79. The static version is doing real work;
the dynamic version is doing about 60% more of it, by this measure. Query-wise and
key-wise branches contribute nearly as well on their own as together, and post-compose
(on the weights) alone beats pre-compose (on the scores) alone, 11.05 vs 11.54 — the low-
rank projection branches (B2/B4) matter more than the gates (B3/B5).

## Why it's cheap

The reason DCMHA doesn't cost what a full head-to-head transform would is a
decomposition, not a shortcut. Conceptually, composing every head with every other head
needs an $H \times H$ transform per query/key pair — quadratic in the number of heads.
DCMHA factors that tensor into a query-wise term plus a key-wise term (row + column), and
factors each of *those* into a rank-$R$ product plus a diagonal gate (low-rank + diagonal
decomposition). The cost drops from $H^2$ to $2HR + H$, and — a nice side effect — the
key-wise half can be computed once and cached alongside K/V, which is exactly what a
serving stack needs.

<ComposeCost />

At the paper's own 6.9B-scale example ($D_h = 128$, $R = 2$): roughly 1.3% extra
parameters and 1.9–3.3% extra FLOPs, depending on sequence length. Rank $R = 2$ turns out
to be close to a sweet spot in the ablation ($R{=}1$: 10.87 ppl, $R{=}2$: 10.83,
$R{=}4$: 10.89 — non-monotonic, and not worth pushing higher).

## The headline number

Trained on The Pile, matched Chinchilla-style token budgets, three model families: the
scaling curves show **DCFormer-834M matches a plain Transformer trained with roughly
1.87 times the compute**, and DCFormer++ (RoPE + SwiGLU added to both sides) matches its
own baseline at roughly 1.67 times. That gap doesn't shrink with scale — DCMHA's relative
improvement decays more slowly than the RoPE+SwiGLU improvement does, which is the
favorable direction.

The result that carries the abstract is the 300B-token run against the actual Pythia
suite:

<Figure
  src="/articles/dcformer/fig2.png"
  alt="Line chart with compute on the x-axis (2.8B times 300B, 6.9B times 300B, 12B times 300B tokens) and loss on the y-axis. Teal circles mark actual Pythia models at each compute point, connected by a dashed fitted trend line. Orange stars mark DCPythia at the 2.8B and 6.9B points, each notably lower than the Pythia line at the same compute. Two horizontal arrows connect each DCPythia point across to where the Pythia trend line reaches the same loss, labeled 1.85x and 1.97x respectively."
  caption="DCPythia reaches a given loss at less compute than Pythia needs for the same loss — and the multiplier grows from 1.85× at the 2.8B scale to 1.97× at 6.9B, not shrinks (Xiao et al., 2024, Figure 4)."
/>

<BenchBars
  title="Pile validation perplexity, 6.9B-class models"
  unit=""
  bars={[
    { label: "Pythia-6.9B", value: 6.29 },
    { label: "DCPythia-6.9B", value: 5.95, highlight: true },
    { label: "Pythia-12B", value: 6.01 },
  ]}
/>

DCPythia-6.9B's 5.95 beats Pythia-12B's 6.01 — a model with close to half the
parameters, ahead on the metric that matters for pretraining. It also edges out on
average 0-shot downstream accuracy (56.7 vs 56.5) and 5-shot (57.7 vs 57.2). The gap over
its own size class is not close: Pythia-6.9B sits at 6.29, meaningfully behind.

| Model | Pile ppl | Flan ppl | Avg 0-shot acc |
|---|---|---|---|
| Pythia-2.8B | 6.63 | 8.16 | 53.1 |
| DCPythia-2.8B | 6.36 | 7.68 | 54.5 |
| Pythia-6.9B | 6.29 | 7.85 | 55.1 |
| **DCPythia-6.9B** | **5.95** | **7.13** | **56.7** |
| Pythia-12B | 6.01 | — | 56.5 |

The gap is largest on the Flan Collection (instruction-following/few-shot/CoT data) and
grows with scale, which the authors read as DCMHA disproportionately helping the harder,
more compositional end of the task distribution — a reading the paper backs up with a
purpose-built test.

## A synthetic test built to need composition

The authors built a 74-task, 888-example diagnostic where getting the right answer
requires *simultaneously* attending to the right source token and applying the right
output transformation (e.g., mapping an object to its superclass) — precisely the
combination a head that only ever reads its own fixed QK/OV circuit should struggle with:

<BenchBars
  title="Synthetic composition task — accuracy"
  unit="%"
  bars={[
    { label: "Pythia-6.9B", value: 31.9 },
    { label: "DCPythia-6.9B", value: 39.0, highlight: true },
  ]}
/>

Perplexity on this set drops from 10.05 to 7.36 alongside the accuracy jump — a much
bigger swing than on Pile or Flan, and the paper's own explanation is the one you'd
expect: this task rewards recombining an existing head's QK circuit with a different
head's OV circuit on the fly, which is the one thing static heads structurally cannot do.
The head-diversity analysis (captured variance of concatenated QK and OV projection
matrices, lower meaning more diverse heads) backs this qualitatively too — DCPythia shows
markedly more QK-circuit diversity than Pythia, and moderately more OV-circuit diversity.

## The honest costs

None of this is free, and the paper says so plainly. Composition is I/O-bound, not
compute-bound, and the reference implementation has no fused kernel — plain JAX for
training, plain PyTorch for inference:

| Size | Training throughput (DCFM++ / TFM++) | Inference throughput (DCFM++ / TFM++) |
|---|---|---|
| 2.8B | 74.5% | 81–88% |
| 6.9B | 83.1% | 89–95% |
| 13B | 84.4% | 90–95% |
| 33B | 89.2% | 90–95% |

The overhead shrinks as models scale up, and the authors are explicit that a fused
kernel — FlashAttention-style — is headroom they haven't taken. A separate lever recovers
most of it directly: raising the local-to-global attention ratio and composing only
query-wise (dropping the key-wise branches) pushes DCFormer++-6.9B's training throughput
from 83.1% back up to 92.5% of baseline, at a small, still-net-favorable cost in
perplexity.

<Callout type="note">
Two more honest limits, stated in the paper's own words. First: **DCMHA doesn't
transplant onto a pretrained model.** Continual-pretraining a 1.4B LLaMA-style checkpoint
into a DCFormer for a tenth of its original training steps produced no real improvement —
the composition that matters most happens in early layers, and early-layer gradients are
too small during fine-tuning to move already-settled MHA weights. DCFormer has to be
trained from scratch. Second: the paper is explicit that **matching SOTA was never the
goal** — DCPythia deliberately keeps every other Pythia hyperparameter fixed, to isolate
what DCMHA alone contributes, rather than stacking it with every other efficiency trick
to chase a leaderboard number.
</Callout>

The mechanism transfers outside language too: on ImageNet-1K, DCViT-S/16 at 1.03× the
baseline's parameters (68.0 top-1 at epoch 90) matches ViT-M/16 at 1.72× the parameters
(67.1 top-1) — the same roughly 1.7× parameter-efficiency story, in a different domain,
on one held-out test.

## A different axis from the memory-side attention variants

If you've read the [field guide to attention mechanisms](/articles/attention-mechanisms)
on this site, it's worth being precise about where DCMHA sits relative to that map. MQA,
GQA, and MLA all operate on what that piece calls the **memory axis** — they *share* or
*compress* the K/V heads to shrink the KV cache, trading some quality for less memory
bandwidth at decode time. DCMHA doesn't touch the cache at all: the number of physical
K/V heads is unchanged, nothing shrinks. It operates on an orthogonal axis entirely — not
how many heads you cache, but what each head is allowed to compute, by letting it borrow
another head's QK or OV circuit, per token. A model could in principle combine GQA's
cache savings with DCMHA's composition; the paper doesn't test that combination, so
read it as plausible, not demonstrated. (For the general design question of specializing
attention below the layer level, [HydraHead](/articles/hydrahead) is the other piece on
this site working that seam, from a different angle.) The broader landscape of
architecture choices — attention, position encoding, MoE, diffusion — is mapped at
[/architectures](/architectures).

## How much of this actually caught on

<Callout type="warn">
Being fair to the number in the abstract requires two caveats the paper itself doesn't
hide. The Pythia baselines are **re-run by the authors** under matched settings, not
copied from the original paper — a genuinely controlled comparison, and the authors say
so directly ("our aim is not to obtain SoTA results, but to clearly quantify the gain").
And the compute-equivalence multipliers (1.87×, 1.67×, 1.85×, 1.97×) come from fitting
scaling-law lines to **three data points per curve** — reasonable given the cost of
training each point, but a thinner fit than, say, Chinchilla's own study.
</Callout>

Two years after an ICML oral, roughly a dozen citations and one flagged "influential" is
a modest academic footprint — the kind of number that would normally suggest an idea
that didn't pan out. What actually happened looks different: Caiyun Technology, the
paper's own industry co-author's employer, shipped DCFormer into a production language
model and upgraded an AI-RPG platform to run on it, and the GitHub repository was still
being extended — DeepSpeed ZeRO support, Hugging Face Trainer integration — well into
2025. No successor paper has benchmarked against DCFormer as a state-of-the-art baseline
to beat; the adoption signal here is industrial, not academic. That's a real but
different kind of validation than a citation count measures, and it's the honest way to
read this one: not a paper that changed the field's direction, but a working piece of
architecture that one production system actually adopted, sitting quietly under-cited.

## The take

Fixed, independent attention heads leave two things on the table: a low-rank bottleneck
that a wider head dimension fixes at a real cost, and redundancy nothing forces heads to
avoid. DCMHA's Compose function fixes both by recombining heads' scores and weights
per token, through a decomposition cheap enough that a 6.9B model pays about 1.3% more
parameters for it. The result — DCPythia-6.9B beating Pythia-12B on perplexity at
roughly half the parameters — is real, reproduced by the authors under controlled
settings, and backed by a synthetic test built specifically to need what static heads
can't do. It just hasn't been the paper everyone cites. Production adoption at one
company and a quiet GitHub repository are what two years actually bought it — which is a
fine outcome for a piece of architecture, even if it isn't the one the citation count
would lead you to expect.

---

*Built on [Improving Transformers with Dynamically Composable Multi-Head Attention](https://arxiv.org/abs/2405.08553)
(Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan; Beijing University of Posts and
Telecommunications / Caiyun AI; ICML 2024, Oral) and its
[code release](https://github.com/Caiyun-AI/DCFormer). Figures are the paper's own
Figures 2 and 4, reproduced for commentary. Tables and numbers are the authors' except
where marked as this site's own illustrative simplification (the Compose bar demo, the
head-cost demo); interactive diagrams are mine.*