2026-07-05 · 7 min · llm · mixture-of-experts · attention · inference-optimization · explainer
LongCat 2.0 is Meituan's largest open model: a Mixture-of-Experts language model with 1.6 trillion total parameters that activates only ~48 billion per token, released with MIT-licensed weights on Hugging Face and ModelScope (plus a separate LongCat-2.0-FP8 artifact for deployment). Two things make it worth a close read. First, the whole training run and serving stack were built on AI ASIC superpods rather than GPUs — a claim about frontier-scale training on alternative silicon. Second, its headline architectural bet is LongCat Sparse Attention (LSA), a redesigned sparse-attention indexer aimed squarely at long-context serving.
Where the parameters live
1.6T total but ~48B active is a ~3% activation rate — the sparsity that makes a model this large affordable to run. But LongCat 2.0 has a second, less common parameter store: an N-gram Embedding of 135B parameters, inherited from LongCat-Flash-Lite. Crucially, these are not extra experts — they expand parameters along sparse dimensions orthogonal to the MoE, adding capacity through a token-n-gram lookup rather than a wider expert pool. Meituan frames it as a scaling principle: once MoE sparsity has "crossed the sweet spot," a bounded slice of N-gram parameters beats simply adding equivalent MoE capacity.
Of a 1.6T-parameter model, only about 48B run for any given token — the router lights a sparse handful of experts (~3% of the pool), which is what makes a model this large cheap enough to serve. The 135B N-gram Embedding sits outside that pool: it expands parameters along sparse dimensions orthogonal to the MoE, adding capacity through a token-n-gram lookup rather than more experts. Step the token and both stores light a different sparse subset.
If the MoE routing here is unfamiliar, we built it up from nothing in Mixture of Experts, from scratch — the router, the top-k gate, and why activating a sparse subset is the whole economic argument for a trillion-parameter model.
LongCat Sparse Attention
The expensive part of long context is attention: under full attention every query reads every past token, so a 1M-token context is brutal to serve. The field's fix is to make attention sparse — read only a chosen subset of the past. LongCat's starting point is DeepSeek's Sparse Attention (DSA) and its "Lightning Indexer," whose weaknesses Meituan names directly: output discontinuity and a quadratic scoring bottleneck. LSA answers with three orthogonal improvements.
The core one is Hierarchical Indexing (HI), and it is best understood against the sparse-attention design we covered earlier — MiniMax Sparse Attention (MSA), which scores the past in 128-token blocks and keeps the top-k whole blocks. LSA treats that block selection as only a coarse recall: a cheap block-level pass proposes candidate blocks, then a fine pass selects the most relevant individual tokens inside them. Same idea of scoring first and reading less — but token-precise, over a smaller candidate set the indexer has to score. Flip the mode below to watch the budget move from whole blocks to individual tokens:
Both methods first score the past at the block level (the pips above each block). MSA stops there — it keeps the top-3 whole blocks and attends every token inside them. LSA treats that as a coarse recall, then runs a second, fine pass that keeps only the most relevant individual tokens inside the recalled blocks — token-precise selection for a smaller candidate set to score. The local block (amber) is always kept.
The second improvement, Cross-Layer Indexing (CLI), cuts cost on a different axis. Deciding what to read costs an index pass per layer; but attention saliency is empirically stable across adjacent layers, so LongCat shares one index every 2 layers — half the layers reuse a neighbour's selection instead of recomputing it, taught by cross-layer distillation during training. The same trick collapses the model's 3-step Multi-Token-Prediction draft into a single shared pass for speculative decoding:
Selecting which tokens to read costs an index pass per layer. Because attention saliency barely moves between adjacent layers, LongCat reuses one selection across a group — one index every 2 layers, halving the indexer work, taught by cross-layer distillation during training. The same trick collapses the 3-step Multi-Token-Prediction draft into a single shared pass for speculative decoding. (Note: the SGLang deployment drops the separate hierarchical stage for simplicity — CLI is the part that ships.)
The third, Streaming-aware Indexing (SI), is a memory-systems move: it reshapes the token-selection budget to combine hardware-aligned contiguous access with dynamic random selection, turning fragmented reads into predictable sequential ones for coalesced HBM bandwidth. Together the three attack the same target from different sides — HI shrinks what the indexer scores, CLI shrinks how often it runs, SI makes the reads it does issue cheaper.
One honest deployment note: the public SGLang integration drops the hierarchical stage for simplicity and serves LongCat 2.0 on 16× H20 with tensor + expert parallelism. So the shipping inference path today is the CLI/SI part of LSA, not the full HI pipeline.
Training, and what "1M context" means
Pretraining spans 35T+ tokens with, Meituan reports, no rollbacks or irrecoverable loss spikes — a stability claim about frontier-scale training on ASICs. Long-context ability comes from training on hundreds of billions of tokens of 1M-context data. That "1M" is a training-data figure: the sources describe the data the model saw, and the README does not publish a separate usable-context window or a long-context retrieval eval (e.g. RULER/HELMET) to pin down how far that quality actually holds at inference. Worth keeping the two apart.
The numbers, in full
LongCat 2.0 is evaluated against Gemini 3.1 Pro, GPT-5.5, and three Claude Opus checkpoints (4.6 / 4.7 / 4.8). The official chart:
On SWE-bench Pro LongCat's 59.5 edges past GPT-5.5 (58.6), Gemini 3.1 Pro (54.2) and Opus 4.6 (57.3) — but the newer Opus checkpoints pull ahead (4.7 = 64.3, 4.8 = 69.2):
IFEval shows the opposite shape: LongCat (90.0) trails Gemini 3.1 Pro (96.1) and GPT-5.5 (95.0), but the newest Claude checkpoints regressed here, so LongCat sits above Opus 4.8 (86.0):
The rest of the suite tells the same "competitive, not leading" story. LongCat edges Gemini 3.1 Pro on Terminal-Bench 2.1 (70.8 vs 70.7) and FORTE (73.2 vs 70.3, matching Opus 4.6), leads it on RWSearch (78.8 vs 76.3) and IMO-AnswerBench (81.8 vs 79.5 for GPT-5.5) — yet on each of those a closed model still tops the row (GPT-5.5 = 77.8 on FORTE; 85.3 on RWSearch; Opus 4.8 = 78.9 on Terminal-Bench; Gemini = 90.0 on IMO-AnswerBench, 96.1 on IFEval, 94.3 on GPQA-diamond, where LongCat is 88.9). On BrowseComp it clearly trails (79.9 vs Gemini's 85.9). No single number here is a headline win over the field.
The take
LongCat 2.0's real contribution isn't a benchmark crown — it's the combination: a genuinely large (1.6T) MoE, trained end-to-end on non-NVIDIA silicon, shipped under MIT weights, with a sparse-attention design that is a real step past the DSA/MSA lineage. LSA's three moves are cleanly separated — HI goes finer than MSA's whole-block selection (token-precise, over a coarsely-recalled candidate set), CLI amortizes the indexer across layers, SI makes the memory access regular — and each targets a distinct cost, which is the kind of engineering that turns theoretical sparsity into wall-clock savings. The honest caveats are the usual open-weights ones: the benchmarks are self-run and self-selected; the model is competitive with Gemini 3.1 Pro and GPT-5.5 on coding/agentic tasks but trails Claude Opus 4.8 on most; the striking "1M context" and "millions of accelerator-days" are provider claims, not independently measured; and the part of LSA that actually ships today (in SGLang) is CLI/SI, with the hierarchical stage dropped for simplicity. For a team that wants frontier-scale, open, and long-context — and can run 16× H20 — it's a serious option. As a claim that you don't need NVIDIA to train at this scale, it's the more interesting story.
Built on the LongCat 2.0 release (Meituan, 2026) — GitHub README and Hugging Face model card, MIT license. All benchmark numbers are provider-reported (in-house harness unless marked * = official model report); the interactive diagrams are illustrations of the mechanism, not measured traces. The benchmark figure is reproduced from the model card for commentary.