# LongCat 2.0: a 1.6T open-weights MoE, and the sparse attention behind it

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/longcat-2
> date: 2026-07-05
> tags: llm, mixture-of-experts, attention, inference-optimization, explainer
**LongCat 2.0** is Meituan's largest open model: a Mixture-of-Experts language model with **1.6 trillion total parameters** that activates only **~48 billion per token**, released with **MIT-licensed weights** on Hugging Face and ModelScope (plus a separate `LongCat-2.0-FP8` artifact for deployment). Two things make it worth a close read. First, the whole training run and serving stack were built on **AI ASIC superpods** rather than GPUs — a claim about frontier-scale training on alternative silicon. Second, its headline architectural bet is **LongCat Sparse Attention (LSA)**, a redesigned sparse-attention indexer aimed squarely at long-context serving.

<Callout type="warn">
Every benchmark here is **provider-reported** and, unless marked otherwise, run in-house under Meituan's own harness. On the published suite LongCat 2.0 does **not** top a single benchmark against the full frontier set: it is competitive with — and on a few coding/agentic tasks beats — **Gemini 3.1 Pro** and **GPT-5.5**, while the strongest closed model (usually **Claude Opus 4.8**) leads every row. Read it as a strong *open-weights* model, not overall SOTA. The ASIC training story and "millions of accelerator-days" are provider details we cannot independently verify.
</Callout>

## Where the parameters live

1.6T total but ~48B active is a **~3% activation rate** — the sparsity that makes a model this large affordable to run. But LongCat 2.0 has a second, less common parameter store: an **N-gram Embedding** of **135B parameters**, inherited from LongCat-Flash-Lite. Crucially, these are *not* extra experts — they expand parameters along **sparse dimensions orthogonal to the MoE**, adding capacity through a token-n-gram lookup rather than a wider expert pool. Meituan frames it as a scaling principle: once MoE sparsity has "crossed the sweet spot," a bounded slice of N-gram parameters beats simply adding equivalent MoE capacity.

<ScaleBank />

If the MoE routing here is unfamiliar, we built it up from nothing in [Mixture of Experts, from scratch](/articles/mixture-of-experts-from-scratch) — the router, the top-k gate, and why activating a sparse subset is the whole economic argument for a trillion-parameter model.

## LongCat Sparse Attention

The expensive part of long context is attention: under full attention every query reads every past token, so a 1M-token context is brutal to serve. The field's fix is to make attention **sparse** — read only a chosen subset of the past. LongCat's starting point is [DeepSeek's Sparse Attention (DSA)](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) and its "Lightning Indexer," whose weaknesses Meituan names directly: **output discontinuity** and a **quadratic scoring bottleneck**. LSA answers with three *orthogonal* improvements.

The core one is **Hierarchical Indexing (HI)**, and it is best understood against the sparse-attention design we covered earlier — [MiniMax Sparse Attention (MSA)](/articles/minimax-sparse-attention), which scores the past in **128-token blocks** and keeps the top-k *whole blocks*. LSA treats that block selection as only a **coarse recall**: a cheap block-level pass proposes candidate blocks, then a **fine pass selects the most relevant individual tokens** inside them. Same idea of scoring first and reading less — but token-precise, over a smaller candidate set the indexer has to score. Flip the mode below to watch the budget move from whole blocks to individual tokens:

<LsaIndex />

The second improvement, **Cross-Layer Indexing (CLI)**, cuts cost on a different axis. Deciding what to read costs an *index pass* per layer; but attention saliency is empirically stable across adjacent layers, so LongCat **shares one index every 2 layers** — half the layers reuse a neighbour's selection instead of recomputing it, taught by cross-layer distillation during training. The same trick collapses the model's 3-step [Multi-Token-Prediction](/articles/multi-token-prediction) draft into a single shared pass for speculative decoding:

<CrossLayer />

The third, **Streaming-aware Indexing (SI)**, is a memory-systems move: it reshapes the token-selection budget to combine hardware-aligned **contiguous** access with dynamic random selection, turning fragmented reads into predictable sequential ones for coalesced HBM bandwidth. Together the three attack the same target from different sides — HI shrinks *what* the indexer scores, CLI shrinks *how often* it runs, SI makes the reads it does issue cheaper.

One honest deployment note: the public SGLang integration **drops the hierarchical stage for simplicity** and serves LongCat 2.0 on 16× H20 with tensor + expert parallelism. So the shipping inference path today is the CLI/SI part of LSA, not the full HI pipeline.

## Training, and what "1M context" means

Pretraining spans **35T+ tokens** with, Meituan reports, **no rollbacks or irrecoverable loss spikes** — a stability claim about frontier-scale training on ASICs. Long-context ability comes from training on **hundreds of billions of tokens of 1M-context data**. That "1M" is a **training-data** figure: the sources describe the data the model saw, and the README does not publish a separate usable-context window or a long-context retrieval eval (e.g. RULER/HELMET) to pin down how far that quality actually holds at inference. Worth keeping the two apart.

## The numbers, in full

LongCat 2.0 is evaluated against Gemini 3.1 Pro, GPT-5.5, and three Claude Opus checkpoints (4.6 / 4.7 / 4.8). The official chart:

<Figure
  src="/articles/longcat-2/fig1.png"
  alt="Six grouped bar panels — Terminal-Bench 2.1, SWE-bench Pro, SWE-bench Multilingual, FORTE, RWSearch, BrowseComp — each comparing LongCat-2.0 (green) against Gemini 3.1 Pro, GPT-5.5, and Claude Opus 4.6, 4.7, and 4.8. Bars are close in height across models; LongCat is near the leaders on coding and search panels but not the tallest bar in any panel."
  caption="LongCat 2.0 vs frontier closed models across code-agent and search benchmarks; bars are unlabeled in the source. LongCat is competitive throughout but tops no panel outright (LongCat 2.0 model card, benchmark chart)."
/>

On **SWE-bench Pro** LongCat's 59.5 edges past GPT-5.5 (58.6), Gemini 3.1 Pro (54.2) and Opus 4.6 (57.3) — but the newer Opus checkpoints pull ahead (4.7 = 64.3, 4.8 = 69.2):

<BenchBars
  title="SWE-bench Pro (%) — provider-reported"
  unit=""
  bars={[
    { label: "LongCat 2.0", value: 59.5, highlight: true },
    { label: "GPT-5.5", value: 58.6 },
    { label: "Gemini 3.1", value: 54.2 },
    { label: "Opus 4.8", value: 69.2 },
  ]}
/>

**IFEval** shows the opposite shape: LongCat (90.0) trails Gemini 3.1 Pro (96.1) and GPT-5.5 (95.0), but the newest Claude checkpoints regressed here, so LongCat sits *above* Opus 4.8 (86.0):

<BenchBars
  title="IFEval (%) — provider-reported"
  unit=""
  bars={[
    { label: "LongCat 2.0", value: 90.0, highlight: true },
    { label: "Gemini 3.1", value: 96.1 },
    { label: "GPT-5.5", value: 95.0 },
    { label: "Opus 4.8", value: 86.0 },
  ]}
/>

The rest of the suite tells the same "competitive, not leading" story. LongCat edges Gemini 3.1 Pro on **Terminal-Bench 2.1** (70.8 vs 70.7) and **FORTE** (73.2 vs 70.3, matching Opus 4.6), leads it on **RWSearch** (78.8 vs 76.3) and **IMO-AnswerBench** (81.8 vs 79.5 for GPT-5.5) — yet on each of those a closed model still tops the row (GPT-5.5 = 77.8 on FORTE; 85.3 on RWSearch; Opus 4.8 = 78.9 on Terminal-Bench; Gemini = 90.0 on IMO-AnswerBench, 96.1 on IFEval, 94.3 on GPQA-diamond, where LongCat is 88.9). On **BrowseComp** it clearly trails (79.9 vs Gemini's 85.9). No single number here is a headline win over the field.

## The take

LongCat 2.0's real contribution isn't a benchmark crown — it's the *combination*: a genuinely large (1.6T) MoE, trained end-to-end on non-NVIDIA silicon, shipped under **MIT** weights, with a sparse-attention design that is a real step past the DSA/MSA lineage. LSA's three moves are cleanly separated — **HI** goes finer than MSA's whole-block selection (token-precise, over a coarsely-recalled candidate set), **CLI** amortizes the indexer across layers, **SI** makes the memory access regular — and each targets a distinct cost, which is the kind of engineering that turns theoretical sparsity into wall-clock savings. The honest caveats are the usual open-weights ones: the benchmarks are self-run and self-selected; the model is competitive with Gemini 3.1 Pro and GPT-5.5 on coding/agentic tasks but trails Claude Opus 4.8 on most; the striking "1M context" and "millions of accelerator-days" are provider claims, not independently measured; and the part of LSA that actually ships today (in SGLang) is CLI/SI, with the hierarchical stage dropped for simplicity. For a team that wants frontier-scale, open, and long-context — and can run 16× H20 — it's a serious option. As a claim that you don't need NVIDIA to train at this scale, it's the more interesting story.

---

*Built on the [LongCat 2.0 release](https://github.com/meituan-longcat/LongCat-2.0) (Meituan, 2026) — GitHub README and [Hugging Face model card](https://huggingface.co/meituan-longcat/LongCat-2.0), MIT license. All benchmark numbers are provider-reported (in-house harness unless marked `*` = official model report); the interactive diagrams are illustrations of the mechanism, not measured traces. The benchmark figure is reproduced from the model card for commentary.*