{"query":"attention","count":20,"results":[{"kind":"articles","slug":"ax-k2","title":"A.X-K2: a sparse-attention upgrade that costs nothing, trained natively in FP8","url":"https://ai.thesatyajit.com/articles/ax-k2","field":"Sparse Gated Attention","snippet":"Start from what A.X-K1 already had: Multi-head Latent Attention (MLA) with a head-specific output gate — a learned, input-dependent gate applied to the attention output before the projection, present in every layer throughout pretraining, n","score":2.928866354087048},{"kind":"articles","slug":"sol-attn","title":"Sol-Attn: deciding which attention blocks to skip while you're already streaming them","url":"https://ai.thesatyajit.com/articles/sol-attn","field":"body","snippet":"Video generation has an attention problem that language models mostly don't. A few seconds of video at a useful resolution is a very long token sequence, attention is quadratic in it, and diffusion runs the whole stack dozens of times per c","score":2.8796991181408558},{"kind":"articles","slug":"hydrahead","title":"HydraHead: hybrid attention at the head, not the layer","url":"https://ai.thesatyajit.com/articles/hydrahead","field":"body","snippet":"So the field mixes them — hybrid attention . Almost everyone does it per layer : interleave whole full-attention (FA) layers with whole linear-attention layers at some fixed ratio (3:1, 7:1), sometimes searched with NAS (as in GLM-5). Hydra","score":2.8691084482869895},{"kind":"articles","slug":"attention-mechanisms","title":"A field guide to attention mechanisms","url":"https://ai.thesatyajit.com/articles/attention-mechanisms","field":"A quality move, not an efficiency one: differential attention","snippet":"Not every variant is about cost. Differential attention (Microsoft, 2024) targets a quality failure: softmax spends attention mass on irrelevant tokens because the weights are forced to sum to 1 — the same pressure that creates attention si","score":2.842989495717547},{"kind":"articles","slug":"minimax-sparse-attention","title":"MiniMax Sparse Attention: let each query pick its own blocks","url":"https://ai.thesatyajit.com/articles/minimax-sparse-attention","field":"body","snippet":"The thing that makes long context expensive is that, under full attention, every query token reads every past token . The KV cache that stores those keys and values grows linearly with sequence length, and the attention itself scales with i","score":2.8371341746072023},{"kind":"articles","slug":"vllm","title":"vLLM: what PagedAttention turned into","url":"https://ai.thesatyajit.com/articles/vllm","field":"Six of the attention backends are not attention","snippet":"…shape is the whole argument. There are 23 files in , and among them are , , , , and an directory. None of those compute attention. They are gated delta networks, linear attention, two generations of Mamba, short convolutions, and latent-KV ","score":2.8250344532208937},{"kind":"articles","slug":"instella-moe","title":"Instella-MoE: a 16B MoE that never touches an NVIDIA GPU","url":"https://ai.thesatyajit.com/articles/instella-moe","field":"Gated MLA: a per-token filter on attention","snippet":"Standard Multi-head Latent Attention compresses the KV cache into a small latent vector, then reconstructs keys and values from it — cheap to store, same attention math otherwise. Gated MLA adds one more piece: after attention produces its ","score":2.8160596460474348},{"kind":"articles","slug":"solar-open2-250b","title":"Solar Open 2: Upstage's 250B-A15B hybrid-attention MoE","url":"https://ai.thesatyajit.com/articles/solar-open2-250b","field":"The hybrid attention stack","snippet":"…at 4× is the margin that turns a 1M-token window from \"possible on a rack\" into \"fits alongside the weights.\" If linear attention is new to you, I built the mechanism up in how transformers attention works and the sparse-attention variants ","score":2.798960960969331},{"kind":"articles","slug":"glm-5-3-flash","title":"GLM-5.3-Flash: 45 layers, 11 of them expensive","url":"https://ai.thesatyajit.com/articles/glm-5-3-flash","field":"The stack","snippet":"The announcement says \"linear attention\" and \"sparse attention\" without naming either, but vLLM's recipe does, and the names matter. The linear layers are KDA — Kimi Delta Attention, the gated delta-rule recurrence from the Kimi line — and ","score":2.788159926783613},{"kind":"articles","slug":"sparda","title":"SparDA: a fourth projection that lets sparse attention prefetch its own KV cache","url":"https://ai.thesatyajit.com/articles/sparda","field":"Three speedups, three different baselines","snippet":"That's the mechanism the attention-time breakdown above shows directly: on MiniCPM4.1-8B at batch 4, per-layer attention time splits into block-selection (green) and block-sparse-attention (blue). In prefill, selection grows with sequence l","score":2.7640656792585476},{"kind":"articles","slug":"sana","title":"Sana: the autoencoder does more of the work than the linear attention","url":"https://ai.thesatyajit.com/articles/sana","field":"body","snippet":"Sana is usually introduced as the linear-attention image model. The paper is titled Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer ; the repo's own summary leads with \"Linear Attention: Replace vanilla attention","score":2.753301874780462},{"kind":"articles","slug":"sana-video2","title":"SANA-Video 2.0: keeping video attention linear without losing the picture","url":"https://ai.thesatyajit.com/articles/sana-video2","field":"The take","snippet":"--- Source: SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation (Chen et al., NVIDIA, 2026), the project page, and the SANA repository. The teaser and architecture figures and the sample clip are ","score":2.7517408804145385},{"kind":"articles","slug":"longcat-2","title":"LongCat 2.0: a 1.6T open-weights MoE, and the sparse attention behind it","url":"https://ai.thesatyajit.com/articles/longcat-2","field":"LongCat Sparse Attention","snippet":"The expensive part of long context is attention: under full attention every query reads every past token, so a 1M-token context is brutal to serve. The field's fix is to make attention sparse — read only a chosen subset of the past. LongCat","score":2.745318122195148},{"kind":"articles","slug":"motif-2-6b","title":"Motif 2.6B: differential attention and PolyNorm, trained at scale","url":"https://ai.thesatyajit.com/articles/motif-2-6b","field":"The block","snippet":"Motif is a 32-layer, hidden-size-2048 dense Transformer — 16 attention heads, no GQA (16 KV heads), a 219,520-token vocabulary, RoPE with θ = 500,000. Standard pre-norm skeleton. What's swapped in are the two coloured boxes: the attention s","score":2.7358192433996305},{"kind":"articles","slug":"ling-3-0-flash","title":"Ling-3.0-flash: a 124B open MoE that runs like a 5B and reaches for 1M tokens","url":"https://ai.thesatyajit.com/articles/ling-3-0-flash","field":"Gated MLA: one full-attention layer per group for exact recall","snippet":"Every group's sixth block is a Gated MLA layer — Multi-head Latent Attention with RoPE and a learned sigmoid gate. MLA compresses keys and values into a low-rank latent before attending, which shrinks the KV cache of the full-attention laye","score":2.7302930793950972},{"kind":"articles","slug":"kda-half-life","title":"KDA has a half-life: linear attention forgets like a radioactive isotope","url":"https://ai.thesatyajit.com/articles/kda-half-life","field":"body","snippet":"Here is a small thing I keep turning over. The forgetting mechanism inside Kimi Delta Attention — the linear attention in Kimi K3 — is the same mathematics as radioactive decay . Not \"reminiscent of\", not \"a useful analogy\". The same two-li","score":2.712761781588087},{"kind":"articles","slug":"goat-optimal-transport-attention","title":"GOAT: entropic optimal transport, minus the transport","url":"https://ai.thesatyajit.com/articles/goat-optimal-transport-attention","field":"body","snippet":"GOAT — Generalized Optimal transport Attention with Trainable priors — says it redefines attention through entropic optimal transport and \"materializes a solution\" for attention sinks. Both halves of that sentence are true. What surprised m","score":2.7050180930623124},{"kind":"articles","slug":"qwen3-8-flash-next","title":"Qwen3.8-Flash-Next: four changes and an honest report","url":"https://ai.thesatyajit.com/articles/qwen3-8-flash-next","field":"QSA: the selector is the cost","snippet":"The global-attention layers use Qwen Sparse Attention . The observation behind it: sparse attention fixes the number of positions read, so the attention itself goes flat in context — but the indexer that chooses those positions still scores","score":2.699222378792234},{"kind":"articles","slug":"intern-s2-mobius","title":"Intern-S2-Mobius: a 35B model that separates memory from reasoning","url":"https://ai.thesatyajit.com/articles/intern-s2-mobius","field":"Forty layers, four memory banks","snippet":"One more thing falls out of matching two arrays in the same config: cycles linear-attention, linear-attention, linear-attention, full-attention every four layers ( ), the same period as the memory-bank assignment. Bank 3 is always the one f","score":2.688585136355083},{"kind":"articles","slug":"chimera-diffusion","title":"Chimera: unbundling RoPE into a diffusion Transformer that extrapolates 6× on video","url":"https://ai.thesatyajit.com/articles/chimera-diffusion","field":"Zero-shot length extrapolation: NoPE's actual payoff","snippet":"This is the sibling result to SANA-Video 2.0, the other linear-attention video approach covered here, and the two make an interesting contrast. SANA-Video 2.0 keeps the same 3:1 linear-to-global attention idea and Block Attention Residuals ","score":2.6806167296524905}]}