Solar Open 2: Upstage's 250B-A15B hybrid-attention MoE

Solar Open 2 is Upstage's newest open-weight model: a 250B-parameter Mixture-of-Experts that activates 15B per token, ships on Hugging Face, and serves a 1M-token context. Upstage frames it as an agentic specialist — built for tool calling, multi-step reasoning, and document-heavy officework — and as a sovereign-AI play, strong in Korean and Japanese as well as English. What makes it worth a close read is not the parameter count but the architecture: instead of a conventional softmax-attention transformer, Solar Open 2 runs a hybrid stack that interleaves one softmax-attention layer with three linear-attention layers, and drops positional encoding entirely.

I read the model card and the accompanying technical report for this. Two framing notes before any benchmark chart: every number below is self-reported by Upstage on its own harness, and the comparison set (DeepSeek-V4-Flash, MiMo-V2.5, Command A+, and others) is Upstage's chosen field. I keep both caveats attached throughout.

The Solar name, and what this is not

If you know the Solar name, you probably know it for depth up-scaling — the 2023 trick behind Upstage's original SOLAR 10.7B, where you grow a model by duplicating and stacking layers from a smaller base rather than training a new shape from scratch. It's reasonable to expect the lineage to continue. It doesn't. Solar Open 2's card describes no depth up-scaling. What it describes instead is a selective weight transfer: the model is initialized from its predecessor, Solar Open 1 (102B-A12B), but "only the 2.3% of weights that survive the architectural change are carried over, and everything else is randomly initialized."

That 2.3% is the honest headline of the training story. The architectural change is drastic enough — a new hybrid attention stack, no positional encoding, an expert pool grown from 128 to 320 — that almost none of the old model's weights fit the new shape. What transfers cleanly are the parts the two generations share: the token embeddings and output layer (same 196,608-token tokenizer), and the fragments of attention and MoE that survive. Everything else starts from noise. So this is not a re-skin of Solar Open 1 and not a depth-up-scaled Solar 10.7B — it is a mostly-fresh 250B model that borrows a running start.

The hybrid attention stack

Here is the distinctive move, and the reason the 1M context is affordable. Solar Open 2 has 48 layers, arranged as twelve identical blocks, each block being one softmax-attention layer followed by three linear-attention layers — the pattern the card writes as [Softmax, Linear×3] × 12. So 12 of the 48 layers are softmax; 36 are linear.

The reason this matters is memory. A softmax layer keeps a KV cache that grows linearly with the sequence — every past token's keys and values must be stored so the next token can attend to them. A linear-attention layer does not: it folds the entire past into a fixed-size recurrent state, so its memory is constant no matter how long the context runs. By making three of every four layers linear, Solar Open 2 keeps a growing KV cache on only 12 of 48 layers — "holding long-context memory to roughly a quarter of an all-softmax model of the same shape," per the card.

Toggle between the hybrid stack and an all-softmax baseline, and drag the context length to watch the KV-cache footprint each one carries:

The KV-cache arithmetic is worth doing by hand, because it is the whole efficiency argument. Each softmax layer is grouped-query attention with 8 KV heads and head_dim 128; storing K and V in fp16 costs, per token per softmax layer:

Multiply by the number of KV-bearing layers and the context length. At 1M tokens:

• Solar Open 2 (12 softmax layers) → about 48 GiB of KV cache.
• An all-softmax stack (48 layers) → about 192 GiB — exactly 4× more.

That 4× is the margin that turns a 1M-token window from "possible on a rack" into "fits alongside the weights." If linear attention is new to you, I built the mechanism up in how transformers attention works and the sparse-attention variants in MiniMax sparse attention; the KV-cache side of the story is how LLM inference works, and squeezing the cache further is TurboQuant.

Three details make the linear layers actually work at this depth, and the card is specific about all three:

• NoPE — no positional encoding. Because the linear layers "encode token order intrinsically in their recurrent state," Upstage removes rotary encoding entirely. The upside the card claims: no RoPE extrapolation limit, so the trained window is not tied to a length distribution seen during training.
• KDA with negative eigenvalues. The linear layers use Kimi Delta Attention (the Kimi Linear lineage — see Kimi K3), but with allow_neg_eigval=True, widening the state-transition write strength to $\beta = 2\sigma(\cdot) \in (0, 2)$. Standard linear cores restrict eigenvalues to $[0,1]$ (decay or persist only); allowing the sign to flip restores the ability to erase and self-correct — the card ties this to genuine state-tracking (parity, modular counting).
• A sigmoid output gate on the softmax layers, which the card says suppresses the "attention sink" pathology and improves long-context extrapolation.

One ordering detail separates it from its cousins: within each block the softmax layer comes first (S-L-L-L), unlike the linear-first ordering (L-L-L-S) of Kimi Linear and Qwen3.5. Upstage's own architecture figure lays the whole thing out — the 12× block on the left, and insets for the MoE, the GQA softmax layer, and the KDA linear layer, color-coded by which weights transferred from Solar Open 1:

expand

Where the parameters live

The sparsity is the economic argument, so account for it. Solar Open 2 is 250B total, 15B active — a 6% activation rate. Each MoE block holds 321 experts: 320 routed plus 1 shared; the router keeps the top-8 routed experts per token, and the shared expert always runs, so 9 experts fire per token. There are no dense layers — every block is MoE. The backbone it inherits from Solar Open 1 is 48 layers, hidden size 4096, head dim 128, 64 query / 8 KV heads, and the 196,608-token vocabulary.

If MoE routing is unfamiliar, I built the router, the top-k gate, and the sparsity argument from nothing in Mixture of Experts, from scratch — the same machinery that lets a 250B model serve at roughly the cost of a 15B dense one.

Training

Upstage is thinner on the data story than on the architecture, and I won't pad it. The concrete figures: ~12 trillion pre-training tokens, on NVIDIA B200 GPUs, for 2M GPU-hours, initialized by the 2.3% selective transfer above. The technical report adds that the run "maximizes value per token over a globally deduplicated corpus" and trains on "purpose-built long-horizon agent scenarios" spanning conversational tool use, coding, and officework — the agentic focus is baked into the data, not just the eval suite.

The tokenizer is the other inherited advantage. Solar Open 2 reuses Solar Open 1's Korean-efficient byte-level BPE tokenizer unchanged; the report claims global-model tokenizers spend 1.2–1.9× more tokens on the same Korean text. In long agent trajectories, where the working context accumulates over many turns, fewer tokens per unit of Korean text translates directly into lower inference cost and a longer effective window — a real, if narrow, edge.

The benchmarks

Upstage's headline figure bundles six benchmarks across knowledge/reasoning and agentic/professional work, with Solar Open 2 (dark violet) against Solar Open 100B and the sub-320B open field:

expand

The clearest win is agentic tool-use in the APEX-Agents suite, where Solar Open 2 roughly doubles the rest of its bracket — a result consistent with the agent-scenario training:

On coding it edges the field on LiveCodeBench v6 (92.4 vs DeepSeek-V4-Flash's 92.3) but sits behind on the harder SWE-Bench Verified agentic coding task — the "competitive, not leading" pattern in miniature:

The full English suite, with the best value in each row bolded (Upstage's own marking):

The shape is consistent: Solar Open 2 leads on MMLU-Pro, LiveCodeBench, and APEX-Agents, and is otherwise a close second to DeepSeek-V4-Flash — which, at 284B-A13B, is a comparable-scale sparse model. Against the smaller Solar Open 100B, the jump is large and uniform (SWE-Bench Verified 15.4 → 70.4, APEX-Agents 2.4 → 16.6), which is the more meaningful comparison since it isolates a generation of progress on one team's harness.

Where Solar Open 2 actually leads is Korean — unsurprising given the tokenizer and data focus. It tops CLIcK, HAE-RAE, KBank-MMLU, KBL, and Ko-GDPval, beating even the closed GPT-5.4 mini and Claude Haiku 4.5 on several:

† Upstage in-house benchmarks — least comparable across vendors. The report's boldest claim rides on the last row: on Ko-GDPval, a Korean officework-agent benchmark, it says Solar Open 2 "essentially matches DeepSeek-V4-Pro (1.6T) at less than a sixth of its size." That is an in-house benchmark and a self-comparison, so weight it accordingly — but the direction (a Korean-specialized 250B beating much larger generalists on Korean agentic work) is plausible and repeated across the daggered rows.

Running it

The weights are ~250B in bf16, so this is multi-GPU territory: Upstage lists a minimum of 4× H200 (141 GB) and recommends 8× H200. The supported production path is vLLM (an Upstage fork), with expert-parallel MoE:

vllm serve upstage/Solar-Open2-250B \
  --served-model-name solar-open2-250b \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --reasoning-parser solar_open2 \
  --tool-call-parser solar_open2 \
  --enable-auto-tool-choice

Solar Open 2 is a reasoning model with a two-position knob: reasoning_effort="high" turns on chain-of-thought (a reasoning block capped at 131,072 tokens), and reasoning_effort="none" answers directly. Because the reasoning trace counts against max_tokens, Upstage recommends leaving room — up to 256K for the full response in high-effort mode — and preserving prior reasoning traces across turns.

from openai import OpenAI
 
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
 
resp = client.chat.completions.create(
    model="solar-open2-250b",
    messages=[{"role": "user", "content": "Prove that the square root of 2 is irrational."}],
    reasoning_effort="high",
    temperature=1.0,
    top_p=1.0,
    max_tokens=131584,
)
print(resp.choices[0].message.reasoning)  # reasoning returned separately
print(resp.choices[0].message.content)

Two nice touches for agent builders. The same vLLM server exposes both an OpenAI-compatible /v1 endpoint and an Anthropic-compatible /v1/messages endpoint, so Claude Code can point straight at it (ANTHROPIC_BASE_URL=http://localhost:8000) with no proxy, and MCP tools reach the model through the standard tool-calling interface. For smaller boxes, NotaAI publishes official quantized builds — INT4, NVFP4, and an INT4-GlobalPruned variant; the NVFP4 format is the same 4-bit floating layout NVIDIA pushes for Blackwell.

License: open weights, with a name tax

Solar Open 2 is open-weight, not fully open-source. It ships under the Upstage Solar License, and the derivative terms are specific: any model you create, fine-tune, or distill from it must prefix its name with "Solar" (e.g. Solar-MyModel-v1), prominently display "Built with Solar" in public materials, and include a copy of the license. That is looser than a research-only license — commercial use and derivatives are allowed — but it is a branded license, not Apache-2.0. If you plan to build on it, the naming and attribution requirements are load-bearing, not boilerplate.

The take

Solar Open 2's real interest is architectural, not positional. It is a clean, well-documented instance of the hybrid linear/softmax direction — three linear-attention layers for every softmax one, no positional encoding, KDA with negative eigenvalues — that makes a 1M-token context affordable by keeping a KV cache on only a quarter of its layers. Paired with a 6%-activation MoE and a Korean-efficient tokenizer, that is a coherent systems story aimed squarely at long-horizon agents, and the honest, unusual init note (2.3% of weights transferred, the rest random) is a refreshing departure from the Solar brand's depth-up-scaling past.

The caveats are the standard open-weights ones, stated plainly. Every number is Upstage's own harness against a field it chose, and on that field Solar Open 2 is a consistent second to DeepSeek-V4-Flash on English work — leading its bracket on a few benchmarks (MMLU-Pro, LiveCodeBench, APEX-Agents) but not the pack. Its clearest edge is Korean, much of it measured on in-house benchmarks. And the license carries a name-and-attribution tax that Apache-2.0 models don't. For a team that wants an open, agent-capable, genuinely long-context model — especially one working in Korean — and can run 4–8× H200, Solar Open 2 earns a serious look. As the strongest open model at 250B, that title still belongs to the model it keeps finishing behind.

Built from the Solar Open 2 model card and technical report (250B-A15B, hybrid attention, 1M context, Upstage Solar License). All benchmark numbers are Upstage-reported; the two figures are reproduced from Upstage's model card and technical report for commentary. The interactive stack diagram is an illustration of the mechanism — the layer pattern, the KV-cache-bearing layers, and the fp16 KV-cache arithmetic use the published config; the linear-attention state memory is a small constant left out of the readout for clarity.