The encoder–decoder Transformer, from first principles

The Transformer of Attention Is All You Need was a translation model, and it had two halves. An encoder reads the source sentence, German say, and turns it into one vector per source token. A decoder writes the English one token at a time, and at every layer it looks back at the encoder's vectors through cross-attention. Decoder-only language models kept the second half and dropped the first; BERT kept the first and dropped the second. This page is about the whole machine, the two families that made it a general-purpose pretrained model, T5 and BART, and why it mostly lost.

The worked numbers use the paper's base model: width $d = 512$, 8 heads, an MLP of width 2,048, 6 encoder layers and 6 decoder layers.

The problem: one sequence in, another out

A sequence-to-sequence model estimates the probability of a target $y = (y_1, \dots, y_T)$ given a source $x = (x_1, \dots, x_S)$, one token at a time:

The source is known in full before generation starts; the target is not. That asymmetry is the whole design. The source can be read in both directions at once, so the encoder uses full attention; the target must be produced left to right, so the decoder is causal.

Before 2017 both halves were recurrent. Sutskever et al. squeezed the whole source into the encoder's final hidden state, one fixed-size vector. Bahdanau et al. argued that this vector was a bottleneck for long sentences, and let the decoder attend over all of the encoder's states at every step instead. That attention is the ancestor of cross-attention; the Transformer kept it and replaced the recurrence on both sides with self-attention.

The encoder

The encoder is a stack of blocks exactly like BERT's: self-attention with no causal mask, then an MLP, each wrapped in a residual add and a LayerNorm. Source token $i$ attends to every source token $j$, left and right, so after 6 layers each of the $S$ vectors describes its token in the context of the entire sentence. The output is a matrix $H \in \mathbb{R}^{S \times d}$, often called the memory. The encoder runs once per source; nothing in it depends on the target.

The decoder: three sublayers

Each decoder block has three sublayers where an encoder block has two:

1. Masked self-attention over the target so far. The causal mask sets every score from target position $t$ to a later position to $-\infty$, so $y_t$ is predicted only from $y_{<t}$.
2. Cross-attention into the encoder's output. The queries come from the decoder's stream; the keys and values come from $H$:

The score matrix is $T \times S$, one score for every pair of a target position and a source position, and it has no mask: every target token may read the whole source. 3. The MLP, per position, as in any Transformer block.

After the last decoder block, a linear layer and a softmax give the next-token distribution. Three attentions, three shapes:

Every decoder layer cross-attends to the same $H$, the output of the encoder's last layer, not to the encoder layer at the same height. When translating "der Hund bellt" the decoder that has written "the dog" sends a query from its newest position, the keys of "der", "Hund" and "bellt" answer, and the values of the best match, "bellt", flow into the stream that will produce "barks".

Teacher forcing, and what happens at inference

Training uses teacher forcing. The decoder's input is the true target shifted right by one, a start token followed by $y_1, \dots, y_{T-1}$, and its output at position $t$ is scored against $y_t$. The causal mask keeps each position from seeing the answer, so one forward pass computes all $T$ predictions in parallel, each conditioned on the true prefix, and the loss is the summed cross-entropy $-\sum_t \log p(y_t \mid y_{<t}, x)$.

At inference there is no true prefix. The encoder runs once; then the decoder generates a token, appends it to its own input, and runs again, usually with beam search (the paper used a beam of 4). Two things are cached. The decoder's self-attention keys and values grow by one entry per generated token, as in any decoder. The cross-attention keys and values, $H W_K$ and $H W_V$, are computed once per layer from the encoder output and never change. The model is trained on true prefixes and run on its own, possibly wrong, ones; that mismatch is called exposure bias.

Where the weights and FLOPs go

Counting matrix weights only, per layer, in units of $d^2$:

• An encoder layer: attention $4d^2$ and an MLP of width $4d$, $8d^2$, so $12d^2$.
• A decoder layer: self-attention $4d^2$, cross-attention $4d^2$ and the MLP $8d^2$, so $16d^2$.

With $d = 512$, $d^2 = 262{,}144$. Six encoder layers hold 18,874,368 matrix weights, six decoder layers 25,165,824, together 44,040,192; biases and LayerNorms add 98,304 more. The paper shares one embedding matrix between the source side, the target side and the output layer, and its English-German vocabulary has about 37,000 tokens, so the table adds about 18.9 million, for about 63 million in all. The paper's Table 3 lists 65 million for the base model.

The FLOPs are where it gets interesting. A cross-attention's four matrices act on two different streams: $W_Q$ and $W_O$ on each target token, $W_K$ and $W_V$ on each source token, once. So per layer pair:

• a source token passes through the encoder layer, $12d^2$, plus the $K$ and $V$ projections in the decoder layer, $2d^2$: $14d^2$ multiply-adds;
• a target token passes through self-attention, $4d^2$, the cross-attention's $Q$ and $O$, $2d^2$, and the MLP, $8d^2$: also $14d^2$.

Each token touches $14d^2$ of the $28d^2$ weights in a layer pair: half the model. With 6 layer pairs and $d = 512$ that is 22,020,096 multiply-adds, about 44 million FLOPs, per token, source or target. A decoder-only model of the same parameter count runs every token through all of its weights, so it costs twice as much per token. This is the observation in T5: an encoder-decoder with $L$ layers in each stack has about the parameters of a $2L$-layer language model and about the compute of an $L$-layer one.

T5: every task as text to text, and span corruption

T5 (Raffel et al., 2019) turned the translation machine into a general pretrained model by making every task text in, text out. A prefix names the task: "translate English to German: …", "summarize: …", "cola sentence: …". Classification becomes generating the label's word. One model, one loss, one decoding procedure.

Its pretraining objective is span corruption. Pick 15% of the tokens, in spans averaging 3 tokens, replace each span with a single sentinel token, and train the decoder to output only the missing spans, each after its sentinel. The paper's own example:

• original: "Thank you for inviting me to your party last week."
• encoder input: "Thank you <X> me to your party <Y> week."
• decoder target: "<X> for inviting <Y> last <Z>"

The encoder sees a sentence with holes and full bidirectional context; the decoder writes a short target, so training is cheap. T5's base model mirrors BERT-base in each stack, 12 layers of width 768, about 220 million parameters, and its other choices became common: a 32,000-token SentencePiece vocabulary, pre-norm with a LayerNorm that has no bias, and relative position biases, one learned scalar per head for each of 32 distance buckets, added to the attention logits. The released sizes ran from 60 million to 11 billion. T5 also compared architectures on the same data, and the encoder-decoder with a denoising objective came out best.

BART: corrupt anything, rebuild everything

BART (Lewis et al., 2019) is the same machine, described as a BERT-like bidirectional encoder feeding a GPT-like left-to-right decoder, trained as a denoising autoencoder. Corrupt a document with any noise, and train the decoder to reproduce the original document in full, not just the missing pieces. The paper tried token masking, token deletion, sentence permutation, document rotation and text infilling: spans with lengths drawn from a Poisson distribution with mean 3, each replaced by a single mask token, including spans of length zero, so the model must also work out how many tokens are missing. Text infilling was the most consistently strong, and the large model combined it with sentence permutation, masking 30% of tokens and permuting all sentences. Reconstructing the whole input makes BART's decoder a strong generator: it set new state-of-the-art results on abstractive dialogue, question answering and summarization, with gains of up to 6 ROUGE.

Why decoder-only took over

T5's own comparison favoured the encoder-decoder, and it uses half its weights per token. Decoder-only models won anyway, for reasons that are mostly about generality at scale rather than quality per FLOP:

• No split to choose. Web text has no natural source and target. A decoder trains on all of it with one next-token loss, every position a target, where span corruption grades about 15% of the input.
• The prompt is the task. GPT-3 showed that a large enough decoder does tasks described in its prompt, with no task-specific input format. Once input and output are one sequence, the same weights read the question and write the answer.
• Conversations append. A decoder keeps its key-value cache and adds the new turn. An encoder is bidirectional, so a new turn changes every encoder state; the whole conversation must be re-encoded and every cross-attention memory rebuilt.
• One stack to scale. One kind of block, one cache, one parallelism plan.

A controlled study, Wang et al. (2022), found that after self-supervised pretraining alone a causal decoder generalised best zero-shot, while a non-causal model trained on masked prediction and then multitask-finetuned did best overall.

The encoder-decoder survives where the input really is a different thing from the output. Whisper encodes 30-second chunks of audio spectrogram and decodes text; the Whisper hallucination article edits that decoder's activations. Translation systems such as NLLB-200 keep it. And the encoder half is often used alone: Imagen conditioned its image diffusion on a frozen T5-XXL encoder, and Breeze TTS 2 reads its text through a bidirectional T5Gemma2TextEncoder.

The block itself is the one on the Transformer page, used twice, joined by one extra attention whose queries and keys come from different sequences.