{"query":"attention","count":20,"results":[{"kind":"articles","slug":"ax-k2","title":"A.X-K2: a sparse-attention upgrade that costs nothing, trained natively in FP8","url":"https://ai.thesatyajit.com/articles/ax-k2","field":"Sparse Gated Attention","snippet":"Start from what A.X-K1 already had: Multi-head Latent Attention (MLA) with a head-specific output gate — a learned, input-dependent gate applied to the attention output before the projection, present in every layer throughout pretraining, n","score":3.1446767843912427},{"kind":"architectures","slug":"transformer","title":"The Transformer, from first principles","url":"https://ai.thesatyajit.com/architectures/transformer","field":"Inference and the KV cache","snippet":"Shrinking that cache is what most attention variants since are about. Multi-query attention shares one key and value head across all the query heads, grouped-query attention shares a few, and multi-head latent attention caches one compresse","score":3.116095315117628},{"kind":"articles","slug":"sol-attn","title":"Sol-Attn: deciding which attention blocks to skip while you're already streaming them","url":"https://ai.thesatyajit.com/articles/sol-attn","field":"body","snippet":"Video generation has an attention problem that language models mostly don't. A few seconds of video at a useful resolution is a very long token sequence, attention is quadratic in it, and diffusion runs the whole stack dozens of times per c","score":3.0950403869337726},{"kind":"architectures","slug":"mamba-ssm","title":"Mamba and selective state spaces, from first principles","url":"https://ai.thesatyajit.com/architectures/mamba-ssm","field":"Why hybrids keep some attention","snippet":"A few attention layers fix most of it. In the Mamba-2 paper's 48-layer, 350M model, perplexity is 8.60 with no attention, 8.26 with 6 attention layers and 8.68 for the Transformer++ baseline; about 10% attention was best. At 2.7B, 58 SSD la","score":3.089355996769424},{"kind":"articles","slug":"hydrahead","title":"HydraHead: hybrid attention at the head, not the layer","url":"https://ai.thesatyajit.com/articles/hydrahead","field":"body","snippet":"So the field mixes them — hybrid attention . Almost everyone does it per layer : interleave whole full-attention (FA) layers with whole linear-attention layers at some fixed ratio (3:1, 7:1), sometimes searched with NAS (as in GLM-5). Hydra","score":3.083356722624493},{"kind":"articles","slug":"attention-mechanisms","title":"A field guide to attention mechanisms","url":"https://ai.thesatyajit.com/articles/attention-mechanisms","field":"A quality move, not an efficiency one: differential attention","snippet":"Not every variant is about cost. Differential attention (Microsoft, 2024) targets a quality failure: softmax spends attention mass on irrelevant tokens because the weights are forced to sum to 1 — the same pressure that creates attention si","score":3.0620550919162914},{"kind":"articles","slug":"minimax-sparse-attention","title":"MiniMax Sparse Attention: let each query pick its own blocks","url":"https://ai.thesatyajit.com/articles/minimax-sparse-attention","field":"body","snippet":"The thing that makes long context expensive is that, under full attention, every query token reads every past token . The KV cache that stores those keys and values grows linearly with sequence length, and the attention itself scales with i","score":3.0547432146496143},{"kind":"articles","slug":"linear-attention-state-roundup","title":"Linear attention's memory problem: four answers in one week","url":"https://ai.thesatyajit.com/articles/linear-attention-state-roundup","field":"body","snippet":"Softmax attention keeps every key and value it has seen. Linear attention keeps one matrix. Everything in between is an argument about what that matrix should be. In the first week of October 2026 four answers to that question landed in my ","score":3.0466743610300657},{"kind":"articles","slug":"vllm","title":"vLLM: what PagedAttention turned into","url":"https://ai.thesatyajit.com/articles/vllm","field":"Six of the attention backends are not attention","snippet":"…shape is the whole argument. There are 23 files in , and among them are , , , , and an directory. None of those compute attention. They are gated delta networks, linear attention, two generations of Mamba, short convolutions, and latent-KV ","score":3.043199646282061},{"kind":"articles","slug":"naive-n05-flash","title":"Naive-N0.5-Flash: no full-attention layers, and a runtime built by AI","url":"https://ai.thesatyajit.com/articles/naive-n05-flash","field":"What \"no full-attention layers\" means","snippet":"…leton and removes the one expensive part. The in the config is a 48-entry list of ones and zeros. A is a Sliding-Window Attention layer; a is a DeepSeek Sparse Attention layer. There is no . Counting the list: 39 ones and 9 zeros (measured)","score":3.0431216787300963},{"kind":"articles","slug":"instella-moe","title":"Instella-MoE: a 16B MoE that never touches an NVIDIA GPU","url":"https://ai.thesatyajit.com/articles/instella-moe","field":"Gated MLA: a per-token filter on attention","snippet":"Standard Multi-head Latent Attention compresses the KV cache into a small latent vector, then reconstructs keys and values from it — cheap to store, same attention math otherwise. Gated MLA adds one more piece: after attention produces its ","score":3.0346289515510696},{"kind":"architectures","slug":"kimi-k3","title":"Kimi K3's hybrid block, from first principles","url":"https://ai.thesatyajit.com/architectures/kimi-k3","field":"Attention Residuals: attention over depth","snippet":"…: The report calls that a bottleneck \"reminiscent of RNNs over time\": all prior layers compressed into one running sum. Attention Residuals apply the move the Transformer made along the sequence to depth. Each layer owns a learned pseudo-qu","score":3.0345836646675393},{"kind":"articles","slug":"kvzip-kv-compression","title":"KVzip: compress the KV cache once, by scoring it with the model's own reread","url":"https://ai.thesatyajit.com/articles/kvzip-kv-compression","field":"Attention Matching: throw the pairs away and synthesize a smaller cache","snippet":"… them, and build a smaller set of synthetic keys and values that reproduce what the full cache would have done. That is Attention Matching (Zweiger, Fu, Guo and Yoon Kim; code, MIT). Replace the full cache for one KV head with a shorter cac","score":3.0333779836305395},{"kind":"articles","slug":"solar-open2-250b","title":"Solar Open 2: Upstage's 250B-A15B hybrid-attention MoE","url":"https://ai.thesatyajit.com/articles/solar-open2-250b","field":"The hybrid attention stack","snippet":"…at 4× is the margin that turns a 1M-token window from \"possible on a rack\" into \"fits alongside the weights.\" If linear attention is new to you, I built the mechanism up in how transformers attention works and the sparse-attention variants ","score":3.0144269513324424},{"kind":"articles","slug":"sparda","title":"SparDA: a fourth projection that lets sparse attention prefetch its own KV cache","url":"https://ai.thesatyajit.com/articles/sparda","field":"Three speedups, three different baselines","snippet":"That's the mechanism the attention-time breakdown above shows directly: on MiniCPM4.1-8B at batch 4, per-layer attention time splits into block-selection (green) and block-sparse-attention (blue). In prefill, selection grows with sequence l","score":3.0068959402096946},{"kind":"articles","slug":"glm-5-3-flash","title":"GLM-5.3-Flash: 45 layers, 11 of them expensive","url":"https://ai.thesatyajit.com/articles/glm-5-3-flash","field":"The stack","snippet":"The announcement says \"linear attention\" and \"sparse attention\" without naming either, but vLLM's recipe does, and the names matter. The linear layers are KDA — Kimi Delta Attention, the gated delta-rule recurrence from the Kimi line — and ","score":3.00194558792828},{"kind":"articles","slug":"whistle-stt","title":"Cactus Whistle: speech-to-text in 16.9 MB, and a FLEURS bar the Whisper paper doesn't support","url":"https://ai.thesatyajit.com/articles/whistle-stt","field":"Gated cross-attention, and eight gates that are wide open","snippet":"…out 3 × 10⁻¹⁰ (measured values, reasoned arithmetic). At inference the scalar gate does nothing: Whistle's \"gated cross attention\" is plain cross-attention, the gate having earned its keep in training and saturated. The decoder's self-atten","score":2.996701566216032},{"kind":"articles","slug":"sana","title":"Sana: the autoencoder does more of the work than the linear attention","url":"https://ai.thesatyajit.com/articles/sana","field":"body","snippet":"Sana is usually introduced as the linear-attention image model. The paper is titled Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer ; the repo's own summary leads with \"Linear Attention: Replace vanilla attention","score":2.9708442904189876},{"kind":"articles","slug":"sana-video2","title":"SANA-Video 2.0: keeping video attention linear without losing the picture","url":"https://ai.thesatyajit.com/articles/sana-video2","field":"The take","snippet":"--- Source: SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation (Chen et al., NVIDIA, 2026), the project page, and the SANA repository. The teaser and architecture figures and the sample clip are ","score":2.967379154965309},{"kind":"articles","slug":"glm-5-3-cmp170hx","title":"GLM-5.3-Flash on four mining cards: the hard part is an Ampere kernel","url":"https://ai.thesatyajit.com/articles/glm-5-3-cmp170hx","field":"The real blocker: sparse attention wants Hopper","snippet":"GLM-5.3-Flash has a hybrid attention stack. Of its 45 layers, most use a linear attention variant (a gated-delta recurrence the config calls KDA), and only eleven keep a real KV cache and run DeepSeek-style sparse attention (DSA). I walked ","score":2.967006458730733}]}