Architectures

Read bottom‑to‑top. Kimi K3 stacks a hybrid block — KDA (linear‑time attention), Gated MLA (full attention for exact recall) and Stable LatentMoE (shared + routed experts) — into the residual stream. The twist is Block Attention Residuals: every residual add pulls a gated read (α) off a bus carrying every earlier block’s output, not just the layer below — a learnable skip across the whole depth.

Instead of attending over the whole sequence, Mamba scans it: a single fixed-size state is carried step to step by a linear recurrence, and the transition parameters (Δ, B, C) are computed from the input — so the model can selectively remember or forget while staying linear in length.

Each block refines the token stream twice — self-attention lets positions mix, the MLP thinks per-token — and both sub-layers are wrapped in a normalize → transform → add residual, so the original signal (and its gradient) flows straight up through all N stacked layers.

The queries never change — only the Key/Value side collapses. Going MHA → MQA → GQA → MLA folds many KV heads into few, or into a single compressed latent, shrinking the per-token KV cache that dominates memory at long context while keeping quality nearly intact.

Only the top-2 routed experts (plus an always-on shared expert) fire for any given token, so a layer can hold a huge pile of parameters while each token pays the compute of just a few small FFNs.

A Diffusion Transformer denoises image latents by treating patches as tokens: a stack of blocks predicts the noise ε, while the timestep and caption are pooled through a small MLP that adaptively modulates each block’s norm (adaLN). In the MMDiT variant, image and text tokens keep separate projections but mix through one joint self-attention.

A looped Transformer runs one shared block over and over — the output of a pass is fed back as the next pass’s input — so it gets the effective depth of a much taller model while storing the parameters of just a single layer. The extra reasoning is paid for in compute, not weights.

Unlike a left-to-right autoregressive model that commits one token at a time, a masked-diffusion LM starts from an all-[M] sequence and, under bidirectional attention, unmasks a few tokens each step and re-predicts the rest — refining the whole sentence in parallel over T steps.

RoPE encodes a position by rotating each pair of feature dimensions by an angle proportional to that position. Because a dot product only cares about the angle between two vectors, attention ends up reading the relative distance (m − n) for free.

A router at every layer keeps only a fixed fraction of tokens for the block; the rest skip it through the residual connection. Because the selected set changes layer to layer, each token gets a different effective depth — and the network spends compute only where it helps.

BERT is a Transformer encoder: with no causal mask, every position attends to the whole sequence at once, so the hidden state for [MASK] is built from context on both sides. The MLM head then reads that position and predicts the missing word.

The encoder reads the whole source at once and produces a set of vectors; the decoder generates the target left-to-right, and at each step its cross-attention queries that encoder output — the one wire that turns two stacks into a translator.