Search

…the whole thing: Three pieces do all the work: the IDF weight, the saturating term-frequency factor (the part), and the length normalization (the part). Here they are, color-coded — the map for the rest of the piece: Take them one at a time

The BM25 walkthrough has the whole derivation; the code is a direct transcription of it. Lucene's non-negative IDF, term-frequency saturation at , length normalization at :

…se-dives, entropy crashes, KL explodes. The IS weight was never optional; it's the approximation's load-bearing term. - Length normalization is stable but worse. Dividing the objective by response length is common (GRPO and CISPO both do it

The token-level loss has a side effect the paper names length inertia . Because the loss is not normalized by length, the model discovers a lazy shortcut: emitting more tokens is mathematically safer, so responses inflate even on easy probl

…urating, or S-shaped as needed, and pick up higher-order interactions a single activation can't. Mold it: The per-power normalization is the load-bearing detail — it's what stops the x³ term from exploding the activation scale, which is exa

…oning chains far longer than the task needs, which inflate latency and serving cost for no accuracy gain. Hybrid Median-length Policy Optimization (HMPO) is a single-stage token-efficiency method with a neat trick for the length budget: don

…nutes , a reproducible speaker fine-tuning protocol, and vocoder super-resolution to 48 kHz . It also handles hard text-normalization cases and degraded reference speech without a separate cleanup stage.

…ter and how long to draft. Deeper drafters help up to a point (the released config uses three MoE layers), and accepted length keeps rising with proposal length where a fully-parallel DFlash block would have decayed — which is the whole arg

… that re-weights each head before the shared output projection. It's a small module, but it's load-bearing — remove the normalization and RULER's extended-context score drops from 87.5 to 71.4 . A learnable scale also beats a learnable gate

…r contexts: Push a hidden state through it and read it out on the vocabulary: where is the unembedding and is the final normalization. Turn that around and every token gets a direction . For token , its J-lens vector is the row of — the dir

… The latent is injected into every decoder layer through adaptive-RMS norm — the latent sets the per-layer scale of the normalization, which is a cheap, stable way to condition a deep stack on a global summary (the same conditioning trick d

…ion (BD3-LM): generate fixed-size contiguous blocks left-to-right, with diffusion inside each block. That buys variable length and a per-block KV cache, but the block is rigid — it can only extend left-to-right, so no infilling, and the cac

Toggle between the hybrid stack and an all-softmax baseline, and drag the context length to watch the KV-cache footprint each one carries: The KV-cache arithmetic is worth doing by hand, because it is the whole efficiency argument. Each sof

…e is computed on logits , omitting the softmax. That avoids gradient pressure leaking onto unrelated tokens through the normalization. 4. Two-part regularization. The tokens it means to move — chosen and rejected — are allowed to travel fre

…compact latents by Mage-VAE; and — the key move — images of any resolution and aspect ratio are flattened into variable-length token sequences and packed together with the text tokens in one batch. Per-sample 2D rotary embeddings and variab

Because the state is constant-size, KDA runs in linear time and constant memory in the sequence length — so five of every six layers pay no growing-cache cost at all. Only the single Gated-MLA layer per group keeps a real KV cache, so the m

…el call inside it stays comfortably in-distribution. Zhang calls this property locally in-distribution (LID) . Drag the length multiplier and watch what it buys you: A base Transformer reads the whole long task in one context window. Past t

…n two stages on a RoPE base of 5e6 . YaRN rescales the rotary position frequencies so positions far beyond the training length stay in distribution instead of aliasing into nonsense. Doing it in two 16x steps rather than one 256x leap keeps

…oken-prediction layer (speculative decoding). With IndexShare, KVShare, and end-to-end training, the average acceptance length rises 20% — from 4.56 to 5.47 tokens per verification pass. More accepted tokens per pass means faster generation

On Vicuna-13B (greedy), EAGLE-3 averages 5.51× speedup across five tasks, peaking at 6.47× on HumanEval with an accepted length of 7.54 — a clear step over EAGLE-2, and multiples over Medusa and vanilla speculative sampling: