Spark-X2.6-1.7B: a million-token window from one full layer in four

mdjsonmcpcopy↗ai

2026-10-31 · 17 min · long-context · sliding-window-attention · kv-cache · reasoning · small-models

Why read this

Solid· top 85%

Reads the 3:1 window/full layout and template-only thinking switch from the files, then measures a needle probe to 65k tokens and 54 reasoning runs on CPU.

• Original analysis
• Runs on a laptop CPU
• Concrete numbers to act on

·LLM architecture·Apache-2.0·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 58 of 100, ranked 445 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Spark-X2.6-1.7B comes with two headline claims: a window of up to a million tokens, and three thinking modes, none, low and high. I wanted to know what each is made of, so I read the repository and then ran the model. Both claims turn out to be plainer than they sound. The window comes from a fixed pattern of three short-window layers to one full layer. The thinking modes are one sentence in the system prompt, plus a prefilled </think>.

Labels used throughout: measured means I ran it or computed it from a file I downloaded, reported means the lab says so and I did not check, reasoned means arithmetic or inference from measured pieces.

What the release is

Spark-X2.6-1.7B is a 1.7B-parameter, text-only, bilingual (English and Chinese) language model published on Hugging Face under the organisation XHToken, whose display name is SparkLLM. The model card calls it the second model of the Spark-X2.6 series after the 4B, and says it keeps exactly the architecture of Spark-X2.5-1.7B: the gains are post-training only. The weights are Apache-2.0; I diffed the repository's LICENSE against the canonical text at apache.org (measured): the only difference is the appendix notice, which reads "Copyright 2026 XHToken". The card's license: apache-2.0 agrees.

Who is SparkLLM? The post I started from describes it as the lab behind the Spark family, and the name overlaps with iFlytek's Spark (Xunfei Xinghuo) models. I could not confirm a link: the organisation page says only that SparkLLM develops "general-purpose foundation models and professional AI agent products", with two harness products called AStudio and Loomy, and nothing in the model card, the config or the licence names iFlytek. I will not assert the relationship. Reported only.

There is no paper. The citation block is a @misc entry titled "Spark-X2.6-4B&1.7B: Advancing the Frontier of On-Device Agents for Real-World Tasks". Everything below comes from the repository files, the benchmark SVG on the card, and the runs I did.

The family on the Hub, from the API listing, is: Spark-X2.6-1.7B, -1.7B-Base, and four derived repositories. Sizes of the files in each, measured from the Hub's file listing:

The INT8 and FP8 headers say what is quantised: 1,438,646,272 parameters are 8-bit, which is exactly the 28 layers of attention and MLP projections (I get the same number below), and the 269,010,944 bf16 parameters are the embedding plus norms and gates. The embedding, 131,072 x 2,048, stays at bf16. That is why an "8-bit" 1.7B model is 1.98 GB and not 1.7 GB, and why Q4_K_M is not much smaller than the INT4 safetensors: at this size the vocabulary matrix is 16 percent of the model.

The layout, read from the config

The repository uses custom code (modeling_spark.py, 483 lines, trust_remote_code=True), so there is no native transformers class. I read the file instead of running it, and re-implemented the forward pass from it (the next section says how). The numbers in config.json:

Parameter count, measured from the safetensors shape tables, matches the header's 1,707,657,216: per layer 3,072 x 2,048 (fused QKV) + 2,048 x 2,048 (output) + 3 x 6,656 x 2,048 (MLP) = 51,380,224; times 28 is 1,438,646,272; plus the 268,435,456-parameter embedding, 28 x 16,384 gate weights, and 116,736 norm weights.

Three details I checked against the code. There is no QK-norm. Grouped-query attention is aggressive: 8 query heads share 2 key-value heads, so keys and values are 512 wide each (2 x 256) against a 2,048-wide query. And each head's attention output is multiplied by sigmoid(g_proj(x)), one scalar per head per token, before the output projection. The gate is computed in float32 and cast back.

How the 1M window is reached

There is no RoPE scaling entry, no YaRN, no linear or sparse attention in the code. The long window comes from two choices that are visible in config.json and modeling_spark.py.

1. Three of four layers cannot see far. A sliding_attention layer masks keys more than 512 positions back. These 21 layers carry local syntax and nothing else, and their cache is bounded.
2. The fourth layer is full attention with a slow, partial rotary. Its rope_theta is 5,000,000 against 10,000 in the window layers, and only a quarter of the 256 head dimensions are rotated (64 of them). The other 192 dimensions carry no position at all. My reading, reasoned and not tested by ablation: the unrotated dimensions let a full layer match on content at any distance, and the slow frequencies keep the rotated ones from wrapping inside the window.

The consequence for memory is arithmetic, reasoned from the config. Per token, one full layer stores K and V for 2 heads x 256 dims in 16-bit: 2 x 2 x 256 x 2 bytes = 2,048 bytes. Seven full layers: 14,336 bytes per token. At 1,048,576 tokens that is 14.0 GiB. If all 28 layers were full, 56.0 GiB. The 21 window layers add a constant 21 x 511 x 2,048 bytes, about 21 MiB.

The decode cost is where this stops looking free. With a 1M context, each new token must read the whole 14 GiB of full-layer cache, against 3.2 GiB of weights, and attention does 7 layers x 8 heads x 256 dims x 2 (QK and AV) x 1,048,576 = about 30 billion multiply-adds per token, to 1.4 billion for all the linear layers. At a million tokens attention is roughly twenty times the arithmetic of the rest of the model, and the cache is about four and a half times the weight traffic (14.0 GiB against 3.2 GiB). The layout fixes capacity. It does not make the full layers linear.

The three thinking modes are a template switch

The card says "three configurable thinking levels, none, low, high". I expected a trained budget or a control token. The mechanism is in chat_template.jinja, and it is smaller than that. I rendered the repository's template with jinja2, and it gives these generation prompts for a user message Q (line breaks added for reading):

none : <｜start▁of▁sentence｜><|System|>
       you are a helpful assistant.<｜end▁of▁sentence｜>
       <｜start▁of▁sentence｜><|User|>Q<｜end▁of▁sentence｜>
       <｜start▁of▁sentence｜><|Bot|></think>
 
high : <｜start▁of▁sentence｜><|System|>
       you are a helpful assistant.
       `thinking_effort` guides on how much to think in your thinking channel
       (not including the response channel), supported values include `low` and `high`.
       Now the system is invoked with `thinking_effort=high`.<｜end▁of▁sentence｜>
       <｜start▁of▁sentence｜><|User|>Q<｜end▁of▁sentence｜>
       <｜start▁of▁sentence｜><|Bot|><think>
 
low  : the same, ending `thinking_effort=low`

• none is enable_thinking=false. The template appends </think> straight after <|Bot|>, so the model starts its turn with the think block already closed. Nothing is trained into the weights that I can see from here; it is the prefill that does it.
• low and high both open <think> and differ only by one sentence in the system message. High is the default. Any other value for reasoning_effort raises an error in the template.

So there is no token budget and no hard cap: "low" is a request, in words, that the model is meant to honour through post-training (the card does not say how it was trained). Whether it does is a measurement, and the one I could make is in the next section. The README's recommended sampling is temperature 1.0, top-p 0.95, no top-k, and its example sets max_new_tokens=131072.

One consequence for anyone serving it: the server must pass enable_thinking and reasoning_effort to the template as chat-template kwargs. The card's client example shows chat_template_kwargs: {"enable_thinking": false} for none. It documents no way to ask for low; that is the template's reasoning_effort variable.

What the card reports

The card ships the benchmark table as an SVG (images/all_benchmark.svg). I extracted its text. All numbers are reported, by the lab, "in thinking mode" with the 1.7B at the high level. A star in the table marks results the lab took from other models' public cards, not its own runs, and a dash marks missing scores.

Two things stand out. The year-over-year jump is almost entirely agent, code and long-context rows: Spark-X2.5-1.7B scored 0.9 on the longest MRCR bucket and X2.6 scores 19.9, with the same architecture. So the 1M window is not new in this release; the model learned to use it, if the numbers hold. And on the benchmarks where it loses (AIME 2026, GPQA Diamond, IFEval) it loses to models of similar size or larger. The Average row is the lab's own. I did not recompute it, and since some competitor cells are copied from other cards, it mixes sources.

The card's inference chart (Mac M5 Pro under MLX, RTX 4090 under llama.cpp) is plotted without a zero baseline, by its own subtitle, and I did not try to reproduce it.

What I ran, and how far to trust it

The model needs trust_remote_code, and I did not execute the repository's Python. Instead I wrote a 150-line forward pass in plain PyTorch 2.14.1 (CPU, two threads of a shared four-vCPU box) from modeling_spark.py and config.json, and streamed the weights from the Hub with HTTP range requests so that nothing but the tokenizer and template touched disk. The prompts are rendered with the repository's own chat_template.jinja and tokenised with its tokenizer.json.

What differs from a reference run, so you can discount accordingly:

• Weights are int8, per output channel, including the tied embedding. A bf16 copy would need 3.4 GB of RAM, more than I could spare. For batches of tokens (prefill) the matmul is bf16 activations against the int8 values with the scale applied after; for single-token decode it is PyTorch's dynamic int8 path, which also quantises the activation per token.
• The cache is stored in bf16, and window layers keep only the last 511 earlier tokens, as the config's window of 512 implies.
• Not validated against the reference code. My checks: the model answers "The capital of Anhui Province is Hefei", and prefilling the same 700 tokens as one chunk or as chunks of 128 gives logits that differ by at most 0.24, with the same argmax. A first version used per-tensor activation quantisation in prefill and differed by 8.4 between the two chunkings, so I threw it and its one partial result away. A subtle bug that leaves the model coherent would not show up in either check.

Speed on this box, measured: decode ran at 5.2 to 5.9 tokens per second; prefill at 112 tokens per second for a 9.9k-token prompt, falling to 66 at 40k and 45 at 65k as the full layers' quadratic term grows. These say nothing about a GPU.

Thinking modes on ten small problems

Ten prompts with known numeric answers (17 x 23, the sum 1 to 100, bat and ball, lily pads, a remainder, a rectangle, the 5-machines puzzle and so on), each ending "Give the final answer as 'Answer: <number>'". One sample per prompt per mode at the recommended temperature 1.0, top-p 0.95, seeds 0 to 9; the token cap was 600 for none, 1,500 for low and 2,500 for high. A run that hit the cap before answering counts as wrong.

Read this as a smoke test: ten easy items, one sample each, so one flipped answer is the noise floor. What I take from it:

• Mode changes length more than accuracy, and mostly through one prompt. The high-mode mean is dragged up by the 5-machines puzzle, where it thought for the full 2,500 tokens and never answered. Without that prompt the high and low means are 137 and 78 tokens. The medians, 78.5 for low and 117.5 for high, are closer to the typical gap.
• None is not "no reasoning". It writes the working out in the answer, 95 tokens on average, and still got the machines puzzle wrong ("100 minutes"). The think block being closed does not stop the model from reasoning in the visible reply.
• Low thinking is terse. On the lily-pad prompt, low wrote "Patch doubles daily, takes 48 days to cover whole lake. ... Half would be after 47 days" and answered 47. High on the same prompt wrote "Half at day24 (since 2^24 half)" and answered 24. Different samples of one model, not evidence about the levels, but it shows what high does not buy on a trap question.
• Low also failed on the machines puzzle (635 tokens, answer 500, after computing 5 correctly once along the way).

Eight multi-step problems

The first set was too easy to separate the modes, so I ran a second, with the same setup and caps (none 700, low 1,500, high 2,500): integers below 100 divisible by 3 or 5 (46), the digit sum of 2^15 (26), a two-train catch-up (300 km), the probability of drawing two blue balls from 3 red and 5 blue (36 percent), the least common multiple of 2 to 10 (2,520), x^2 + 1/x^2 given x + 1/x = 3 (7), five people in a row with two adjacent (48), and three consecutive even numbers summing to 78 (28).

Cost does not order the way the labels suggest: none used more tokens than low (237 against 157 on average), because with the think block closed it writes its working into the answer. High used the most. The single failure is low on the train problem. It wrote that train 2 "has traveled 100*0.75=75 km" when it had not left yet, and answered 120 km instead of 300. High and none got every problem right. Eight problems at one sample each is a tiny test, but it points the same way as the first set: at this difficulty I saw no accuracy gain from thinking, and a token bill for high of 2 to 4 times that of none (3,990 against 1,897 on this set, 3,734 against 953 on the first).

So the claim I can support is narrow: the three settings produce three different token budgets, and the extra thinking did not improve the score on 18 problems that a 1.7B model mostly gets right anyway. The card's own benchmarks (AIME, SWE, BrowseComp) are where thinking should pay, and I did not run those.

Needle probe out to 65k tokens

One synthetic haystack per length: random sentences of the form "The quiet lantern measured the old harbour near the falcon on day 212". Five lines of the form "The secret code for vault Orchid is 240891." are planted at 10, 30, 50, 70 and 90 percent of the way through. After prefilling the haystack once, I ask for each vault in turn from the same cache, with thinking off and greedy decoding, and score exact match of the six digits.

Both misses are the same needle, vault Basalt, planted at the 30 percent mark. At 40k tokens it answered 1696853 for the true code 696853, an extra leading digit. At 65k it answered 104863, which is not close. Each depth carries its own vault name and code, so I cannot separate "30 percent is a weak position" from "this particular code is hard to copy"; the sample is too small to say. The cache sizes are the actual bf16 tensors held after prefill, and they equal the formula above to the byte (65,530 x 14,336 for the seven full layers, plus 21 x 511 x 2,048 for the windows), which is a check on my layout reading and not on the model.

What this does and does not show. Retrieval of a distinctive line through the seven full layers mostly works at 65k tokens, which is about 128 times the 512-token window, so something in those layers is attending far. It is a weak test: the needle is lexically unlike the filler, there are only five needles, no distractors resemble them, and the question is a direct lookup. It says nothing about the reported hard buckets, multi-hop Graphwalks or the 512k-to-1M MRCR, and I did not reach 100k, let alone a million: at the speed I measured, a million-token prefill on this box would take many hours (reasoned: more than 6 hours even at the 45 tokens per second of the 65k run, and the rate keeps falling), and its 14 GiB cache would not fit in the RAM I had.

Where this leaves a reader

• The window is a memory claim first. A million tokens fits in 14.0 GiB of bf16 cache because three layers in four hold 21 MiB however long the context is. That is a real engineering choice, and on the 4B (36 layers, nine of them full, 16 query heads over 4 KV heads) the same pattern holds. It does not make attention cheap: at a million tokens each decoded token still reads the full-layer cache and does about twenty times the arithmetic of the weights.
• The reported long-context jump is post-training. X2.5 and X2.6 share an architecture and the card's own table moves MRCR 512k-1M from 0.9 to 19.9. I could not test that range, so I treat it as reported.
• Thinking is a prompt, so you control it. If you serve the model yourself, enable_thinking and reasoning_effort are chat-template variables. A server that does not forward them runs the default, which is high, and you will see long outputs.
• What I would check next: run the bf16 model with the reference code on a GPU and compare my logits; a needle test with distractor vaults; the 128k bucket on a machine with the memory for it.

The licence is Apache-2.0 with the copyright held by XHToken, checked against the file and not only the front matter. The lab's relationship to iFlytek I could not verify, so I leave it out.