Architectures

Read bottom‑to‑top. Kimi K3 stacks a hybrid block — KDA (linear‑time attention), Gated MLA (full attention for exact recall) and Stable LatentMoE (shared + routed experts) — into the residual stream. The twist is Block Attention Residuals: every residual add pulls a gated read (α) off a bus carrying every earlier block’s output, not just the layer below — a learnable skip across the whole depth.

Instead of attending over the whole sequence, Mamba scans it: a single fixed-size state is carried step to step by a linear recurrence, and the transition parameters (Δ, B, C) are computed from the input— so the model can selectively remember or forget while staying linear in length.

Each block refines the token stream twice — self-attention lets positions mix, the MLP thinks per-token — and both sub-layers are wrapped in a normalize → transform → add residual, so the original signal (and its gradient) flows straight up through all N stacked layers.

The queries never change — only the Key/Value side collapses. Going MHA → MQA → GQA → MLA folds many KV heads into few, or into a single compressed latent, shrinking the per-token KV cache that dominates memory at long context while keeping quality nearly intact.

Only the top-2 routed experts (plus an always-on shared expert) fire for any given token, so a layer can hold a huge pile of parameters while each token pays the compute of just a few small FFNs.

A Diffusion Transformer denoises image latents by treating patches as tokens: a stack of blocks predicts the noise ε, while the timestep and caption are pooled through a small MLP that adaptively modulateseach block’s norm (adaLN). In the MMDiT variant, image and text tokens keep separate projections but mix through one joint self-attention.

A looped Transformer runs one shared block over and over — the output of a pass is fed back as the next pass’s input — so it gets the effective depth of a much taller model while storing the parameters of just a single layer. The extra reasoning is paid for in compute, not weights.

Unlike a left-to-right autoregressive model that commits one token at a time, a masked-diffusion LM starts from an all-[M] sequence and, under bidirectionalattention, unmasks a few tokens each step and re-predicts the rest — refining the whole sentence in parallel over T steps.

RoPE encodes a position by rotating each pair of feature dimensions by an angle proportional to that position. Because a dot product only cares about the angle between two vectors, attention ends up reading the relative distance (m − n) for free.

A router at every layer keeps only a fixed fraction of tokens for the block; the rest skip it through the residual connection. Because the selected set changes layer to layer, each token gets a different effective depth— and the network spends compute only where it helps.

BERT is a Transformer encoder: with no causal mask, every position attends to the whole sequence at once, so the hidden state for [MASK] is built from context on both sides. The MLM head then reads that position and predicts the missing word.

The encoder reads the whole source at once and produces a set of vectors; the decoder generates the target left-to-right, and at each step its cross-attention queries that encoder output — the one wire that turns two stacks into a translator.