The bidirectional encoder (BERT), from first principles

In October 2018 Devlin et al. published BERT, and within a year it was the starting point for most of natural-language understanding. It is not a new block. It is the encoder half of the 2017 Transformer, unchanged, trained on a new objective. What is new is what the model is allowed to see, and how it is taught when it can see everything.

The worked numbers use the paper's two sizes: BERT-base (12 layers, width 768, 12 heads, about 110 million parameters) and BERT-large (24 layers, width 1,024, 16 heads, about 340 million).

Take the mask away

A decoder language model predicts token $t+1$ from tokens 1 to $t$, so its attention carries a causal mask: position $i$ may only attend to positions at or before $i$. Of the $T^2$ query-key pairs, $T(T+1)/2$ survive.

An encoder has no next token to predict, so it keeps all $T^2$ pairs. The attention is the same scaled dot product,

with nothing set to $-\infty$ except padding. Every position's output is built from the whole sequence, left and right, at every one of the 12 layers. In "the cat sat on the mat", the vector for "sat" is shaped by "the cat" before it and "on the mat" after it, in the first layer and in every layer above. A causal model can only ever combine one side of that context.

The price is generation. Under a causal mask a position's representation depends only on what came before it, so a decoder can cache keys and values and append one token at a time. In an encoder, appending a token changes the representation of every earlier token, so there is no cache: producing text one token at a time would mean re-running the whole stack for every new token. BERT is built to read, not to write.

The input: three embeddings, summed

BERT's tokenizer is WordPiece with a 30,000-token vocabulary (the released uncased English checkpoints have 30,522 entries). Every sequence starts with a special [CLS] token, and a [SEP] token ends each segment, so a pair of sentences reads [CLS] A [SEP] B [SEP].

Each position's input vector is the sum of three learned rows:

The token table has one row per vocabulary entry. The segment table has two rows, one for sentence A and one for sentence B. The position table has 512 rows, one per position, learned like any other weight rather than the fixed sinusoids of the original Transformer, which is also why BERT cannot read past 512 tokens: position 513 has no row. The sum is layer-normalised and fed to the stack. The blocks are post-norm, as in 2017: attention, add, LayerNorm, then an MLP of width $4d$ (3,072 in BERT-base) with GELU, add, LayerNorm.

Masked language modelling

With every token visible, next-token prediction is meaningless: the answer is in the input. So BERT hides some of the input and asks for it back. Pick 15% of the positions at random. Of those:

• 80% are replaced by [MASK],
• 10% are replaced by a random token,
• 10% are left unchanged.

At every chosen position the final hidden vector goes through a small head (a dense layer, GELU and LayerNorm) and an output layer tied to the token embedding table, and the loss is the cross-entropy against the original token:

where $\mathcal{M}$ is the set of chosen positions and $\tilde{w}$ is the corrupted sequence. Of all tokens, 12% become [MASK], 1.5% become a random token and 1.5% are untouched but still graded.

The split exists because [MASK] never appears when the model is fine-tuned or used. If every chosen token were masked, the model would learn that only [MASK] positions need a careful prediction, and its representation of ordinary tokens would matter less. With 10% random and 10% unchanged, the model cannot tell which visible tokens are real, so it has to build a good contextual vector for every token. The random tokens are 1.5% of the input, which the paper found does not hurt language understanding.

The cost of the recipe is signal per token. Only 15% of positions produce a loss, about 77 of a 512-token sequence, where a decoder language model gets a loss at every position. The paper trained for 1,000,000 steps of 256 sequences, about 40 passes over its 3.3 billion words of books and Wikipedia, 90% of the steps at 128 tokens and the last 10% at 512 to learn the later position rows.

Next sentence prediction, and why it went

BERT had a second objective. Half the time sentence B really follows A in the corpus, and half the time it is a random sentence from elsewhere; the [CLS] vector feeds a two-way classifier that predicts which. The motivation was tasks like question answering and entailment, which reason about the relation between two texts. The paper's ablation found that removing it hurt QNLI, MNLI and SQuAD.

A year later RoBERTa retrained BERT carefully and found the opposite: removing the next-sentence loss and packing each input with contiguous full sentences matches or slightly improves downstream performance. The authors suspected the original ablation had removed the loss but kept the sentence-pair input format. The BERT paper itself reports 97%-98% accuracy on the task, and ALBERT argued why: a random sentence from another document usually has a different topic, so topic matching alone goes a long way, and the model learns little about coherence. ALBERT replaced it with sentence-order prediction, two consecutive segments in the right or the swapped order.

RoBERTa's other changes were about the recipe, not the block: dynamic masking (a new mask each time a sequence is seen, where BERT had fixed masks, its data duplicated 10 times so each sequence carried 10 masks over the 40 passes), batches of 8,000 sequences, a 50,000-entry byte-level BPE vocabulary, and 160GB of text where BERT had used 16GB. Same architecture, clearly better model: most of BERT's headroom was in the training.

[CLS], pooling and the heads

Pretraining leaves a stack that turns $T$ tokens into $T$ contextual vectors of width 768. Fine-tuning puts a small head on top and trains everything end to end, usually for 2 to 4 epochs:

• Sequence classification (sentiment, entailment): take the final vector of [CLS], call it $C$, and apply one new layer $\text{softmax}(C W^{\top})$ with $W \in \mathbb{R}^{K \times H}$ for $K$ classes. Those $K \times 768$ weights are the only new parameters. [CLS] has no word of its own to represent, and because it attends to every position, the model learns to gather the sequence into it.
• Token classification (named entities, tagging): the same kind of layer on every position's vector.
• Span extraction (SQuAD): two learned vectors $S$ and $E$. The start of the answer is a softmax over $S \cdot T_i$ across positions, the end over $E \cdot T_j$, and the best span maximises $S \cdot T_i + E \cdot T_j$ with $j \geq i$.

With these heads BERT set new state-of-the-art results on eleven tasks, among them a GLUE score of 80.5, 7.7 points above the previous best, and a SQuAD v1.1 test F1 of 93.2.

For a sentence embedding, a vector to compare with cosine similarity, raw BERT is poor: Sentence-BERT found that averaged GloVe word vectors beat its [CLS] vector on semantic similarity. BERT used as a cross-encoder, both sentences in one input, is accurate but has to run once per pair: finding the most similar pair among 10,000 sentences is about 50 million forward passes, some 65 hours. Sentence-BERT fine-tuned the encoder in a siamese setup so that each sentence is encoded once, mean-pooled over its tokens, and compared by cosine: about 5 seconds for the same search. Most embedding models since have the same shape: an encoder, a pooling step, and training on the pooled vectors. LFM2.5-Encoder shows the pattern today: mean-pool the last hidden state, one linear head.

Where the parameters and FLOPs go

Here is BERT-base uncased as released, with its 30,522-entry vocabulary, counted weight by weight. Unlike Llama, every linear layer has a bias.

That is the paper's 110 million. Pretraining adds the MLM head's 622,650 weights (its output matrix is the tied token table, so only a dense layer, a LayerNorm and a 30,522-entry bias are new) and the next-sentence classifier's 1,538. The token table alone is 21% of the model, much more than in a large decoder, because the model is narrow and shallow; ALBERT factorised it into two thin matrices so that it would not grow with the width. BERT-large, counted the same way, is 335,141,888.

FLOPs. The 12 layers hold $12 \times 12d^2 = 84{,}934{,}656$ matrix weights, each used in one multiply-add per token: about 170 million FLOPs per token for a forward pass. Attention's own arithmetic adds $4Td$ FLOPs per token per layer, and since every position attends to all $T$ positions, not just its past, at $T = 512$ that is $4 \times 512 \times 768 \times 12 = 18{,}874{,}368$, 11% more. A causal kernel can skip the upper triangle; an encoder has none to skip, so its attention does about twice the work of a causal one at the same length.

What it is good and bad at

Good at: anything where the whole input is available and the output is a label, a set of spans or a vector. Classification, entity tagging, extractive question answering, reranking and retrieval embeddings all read a finished text, and a bidirectional encoder conditions every token on both sides of it in one forward pass. A 110-million-parameter encoder does these jobs for a small fraction of a generative model's cost, with no decode loop. That is why BERT-style encoders still serve most classification and embedding traffic.

Bad at: generation, for the reason above. Long inputs, in the original: 512 learned positions, and attention cost growing with $T^2$. And its training signal is thin, since 85% of positions produce no loss.

What changed since 2018

• Objectives. Next-sentence prediction is gone everywhere. ELECTRA replaced masked prediction with replaced-token detection: a small generator fills the masks and the encoder judges, at every position, whether a token was replaced, so every token gives a loss. Masking rates went up: Wettig et al. found 40% beats 15% for BERT-large, and ModernBERT trains at 30%.
• Position. DeBERTa disentangles attention into content and relative-position terms and adds absolute positions only just before the output layer; its 1.5B model was the first to pass the human baseline on SuperGLUE, 89.9 against 89.8. DeBERTa-v3 is still a common backbone (GLiNER2.5 runs on mDeBERTa-v3). Newer encoders use rotary embeddings.
• The modern block. ModernBERT (December 2024) is BERT rebuilt with a decoder's parts: pre-norm, no biases, GeGLU, rotary positions, and attention that alternates global layers (every third, RoPE base 160,000) with 128-token sliding-window layers (base 10,000). Trained on 2 trillion tokens with a native 8,192-token context, sixteen times BERT's 512, it comes in two sizes, 149M and 395M, with 22 and 28 layers.
• Decoders turned into encoders. Removing a decoder's causal mask and continuing training with masked prediction turns it into an encoder (LLM2Vec; LFM2.5-Encoder does this). And a masked-diffusion language model is a bidirectional stack of the same kind, trained to unmask at every masking rate from 0 to 1 and then run step by step to generate.

The block is the one on the Transformer page with the mask taken off. Everything distinctive about BERT is in what that makes possible to train, and what it makes impossible to generate.