Underdog Saluki 27B: ISTA's 2-bit file with its scales retuned

mdjsonmcpcopy↗ai

2026-10-10 · 24 min · quantization · qwen · tool-calling · llama-cpp · gguf · benchmarks · on-device

Why read this

High· top 30%

Byte-diffs Saluki against ISTA's IQ2_XS: same codes, retuned scales, 49 FFN tensors demoted. Recomputes the 96% and the 7.5x from their own inputs.

• Original, source-checked analysis
• Runs on a consumer GPU
• Concrete numbers to act on

·Quantization & compression·Apache-2.0·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

3 of 3: Reproduces the headline result, or shows from primary files it is wrong

Can I run it?

3 of 3: Open, permissive, runs on reader hardware with instructions

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

3 of 3: The only place this analysis exists

Score 71 of 100, ranked 89 of 494 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Underdog's launch post said three things in one breath: Qwen3.8-27B, "almost 7x smaller", keeps "96% of its benchmark performance" and "beats Qwen at tool calling". A 27B model in under 8 GB is about 2.3 bits per weight. At that width every quantizer I know of loses something real, and tool calling is usually one of the first things to go, because a single wrong token in a JSON argument fails the whole call. So when a 2-bit build claims to beat the full model there, either somebody has a new quantizer, or the comparison is doing some of the work.

It turned out to be neither, exactly. The model card says plainly that Saluki is built on someone else's 2-bit file. The interesting question became what Underdog did to that file, and that is something you can read straight out of the bytes.

expand

What is in the file

The repository is ConwayResearch/Underdog-Saluki-27B-1.0 (ConwayResearch is Underdog's organisation on Hugging Face; its NOTICE reads "Underdog (ConwayResearch)"). It holds one GGUF, Underdog-Saluki-27B-1.0-IQ2-mix.gguf, at 7,898,369,152 bytes, plus two optional vision projectors and an Apache 2.0 LICENSE and NOTICE.

I pulled the first 24 MB of the file with an HTTP range request and parsed the GGUF header. It lists 851 tensors and 26,895,998,464 weights: the text decoder, the token embedding and the LM head, with no vision tower and no multi-token-prediction head. The tensor data adds up to 7,887,374,336 bytes, which is 2.346 bits per weight; with the 11 MB header (most of it the tokenizer) the whole file is 2.349. Against ISTA's BF16 reference at 53.8 GB that is 6.8x smaller, so "almost 7x" is fair.

Then the metadata. general.name is Qwen3.8-27B GSQ-RCO. general.quantized_by is ISTA DASLab. The description field reads "GSQ-RCO non-uniform mixed-precision GGUF (Deep Algorithms and Systems Lab, ISTA)". Underdog didn't hide this. The card's front matter lists ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF as a base model, and the NOTICE file says it outright: "Underdog Saluki 27B 1 is built on this release", naming the file Qwen3.8-27B-GSQ-RCO-IQ2_XS.gguf. The launch post and the X thread never mention it, which is a shame, because the base is the most interesting part of the story.

Only three fields in the metadata differ from ISTA's file: general.file_type went from 19 to 20, general.quantization_version moved, and the importance-matrix path became /root/work/imatrix-qwen3.8-27b.gguf, which is the same file name ISTA ships in its repository. Those are the fields llama-quantize writes. So somebody ran ISTA's file back through the stock quantizer. That much is plumbing. What came out of it is not.

What ISTA built, and why it is already good

ISTA-DASLab's release (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) uses two of their own methods. The file formats are llama.cpp's stock ones, so the cleverness is in which values go into them.

A llama.cpp IQ2 tensor is stored in super-blocks of 256 weights. Each super-block holds one fp16 scale d, a handful of small sub-block scales, and codes that index a fixed grid of points (for IQ2_XXS, points on the E8 lattice, eight weights at a time, with sign bits). The 66 bytes of an IQ2_XXS block are the 2-byte scale plus 64 bytes of codes, which is where its 2.0625 bits per weight comes from. Every format in the file follows the same pattern: a few bytes of continuous scale, and the rest discrete choices.

The usual way to fill those codes is rounding, guided by an importance matrix: GPTQ-style error feedback, or llama.cpp's own search for the grid point nearest each group of weights. GSQ (Gumbel-Softmax Quantization) treats the code choice as something to learn instead. Each coordinate gets logits over nearby grid points, a Gumbel-Softmax relaxation makes the choice differentiable, and the logits and the per-group scales are trained together, block by block, against the full-precision model's outputs. At higher bit widths it only learns a shift of up to two grid steps away from the GPTQ starting point, which keeps the number of logits per weight at five.

expand

RCO then decides how many bits each tensor gets. With every tensor quantized at every candidate type, it searches for one type per tensor so that the file hits a size budget exactly, optimising directly on the task loss. ISTA ships the result as a text file per build. For the 2.50 bits per weight build, the histogram runs from 31 tensors at IQ1_M and 36 at IQ1_S up to 11 at IQ4_XS and 4 at Q4_K. The token embedding gets IQ1_M (1.75 bits; presumably because a lookup only ever reads one row, so its errors stay local), the LM head gets IQ4_XS, and the Gated DeltaNet ssm_alpha and ssm_beta gates stay in BF16.

ISTA's own table shows why Underdog picked this file. At 8.4 GB, IQ2_XS averages 86.03 across AIME25, GPQA-Diamond and LiveCodeBench v6 against 91.87 for BF16, while Unsloth's UD-IQ2_S at the same size averages 78.31. On AIME25 alone the gap is ten points: 96.67 against 86.67.

expand

The site has looked at this family before. Mach-2 Additive used a GSQ-RCO build as its "traditional quant" foil, and runtime dynamic compression compared against GSQ-RCO on the same dense 27B. What nobody had done was take one of these files and keep working on it.

Two files, byte by byte

Both files are public, so I compared them tensor by tensor. Same 851 names, same shapes. 705 tensors kept their type; 146 changed. The changes fall into three groups:

Together that is 524,472,320 bytes, which is what takes ISTA's 8.42 GB to Saluki's 7.90 GB and under the 8 GB line on the launch card.

The 705 tensors that kept their type are where it gets interesting. If Underdog had simply re-run llama-quantize, those would be bit-identical: when the target type equals the source type, the quantizer copies the bytes through. Some are. The token embedding, all 278 MB of it, matches byte for byte at every offset I sampled, and so does output_norm. But the first tensor I tried after those, block 0's attn_qkv, did not match.

So I looked at where in each 256-weight block the bytes differ. For IQ3_S, IQ2_XS, IQ2_XXS, IQ1_S and IQ4_XS it is bytes 0 and 1, and nothing else. Those two bytes are the fp16 super-block scale d. For Q2_K it is bytes 80 to 83, which are d and dmin. For IQ1_M, whose scale is split across the high nibbles of its sub-scale field, it is bytes 49, 51 and 53. In every format, the codes are untouched and the scale moved.

# for one tensor present in both files at the same type
a = fetch(SALUKI, off_s, nblocks * bb).reshape(-1, bb)   # uint8, one row per block
b = fetch(ISTA,   off_i, nblocks * bb).reshape(-1, bb)
print(np.nonzero((a != b).any(0))[0])                    # IQ2_XS -> [0 1]; Q2_K -> [80 81 82 83]
d_s = a[:, 0:2].copy().view(np.float16).astype(np.float64)
d_i = b[:, 0:2].copy().view(np.float16).astype(np.float64)
print(np.median(np.abs(d_s / d_i - 1)))                  # ~0.01

I ran that over a 1,000-block window of every one of the 352 quantized tensors whose type didn't change. Not one code byte differs anywhere. About 98% of block scales moved (the median tensor), by a median of 1.0%, with the 99th percentile around 4%. The changes are centred on zero and look like the residue of an optimiser, not a rescale. In Q2_K blocks, d and dmin moved by the same factor, with a correlation of 0.9997 between their ratios, so one multiplier per block was learned and applied to both. The only tensor whose scales didn't move is the token embedding.

The continuous tensors moved too. Every attn_norm and post_attention_norm in all 64 blocks differs from ISTA's, by about 3e-4 relative. The DeltaNet convolution weights differ by about 0.8%. The 96 ssm_alpha and ssm_beta tensors are now Q8_0, and they sit 1.5 to 2% away from ISTA's BF16 values, against 0.5 to 0.6% for a clean Q8_0 rounding of the same numbers. They were changed in full precision first and quantized afterwards.

This is training. The codes, 79% of the model's weights by count (the 19.96 billion in same-type quantized tensors plus the 1.27 billion in the untouched embedding), are frozen exactly as ISTA's search left them. Everything continuous around them was optimised.

Switch the grid between the two files and the re-typing has an obvious shape. None of it touches the attention or DeltaNet mixers. Almost all of it is FFN down: 41 of 64 ffn_down tensors went to IQ2_XXS, dropping that role from 2.545 to 2.127 bits per weight, plus six ffn_up and two ffn_gate. What surprised me is where it lands. RCO had spent some of its most expensive types on the late ffn_down projections, IQ3_S and IQ4_XS in blocks 50 to 57, and Underdog flattened most of those to the cheapest 2-bit type in the file. A gradient-based search said those tensors were sensitive. Underdog overrode it, presumably to hit the size target with one blunt rule, and then trained to recover the damage. There is no ablation that separates the two, so I can't tell you how much the demotion cost or how much the tuning bought back.

The recipe is in GSQ's own appendix

Freezing the codes and training only the scales is not a trick Underdog invented. It is the last stage of GSQ. Appendix A.2 of the GSQ paper, "Effect of end-to-end scale-only fine-tuning", describes it: "the discrete assignments found by block-wise optimization are kept fixed, and only the per-group scales are updated using the distillation objective". On 2-bit Llama-3.1-8B-Instruct it lifts the five-task zero-shot average from 65.54 to 68.55, plus 3.01 points, from a single epoch over 4,096 sequences.

It is cheap for a reason. There are 99,993,600 super-block scales in Saluki's quantized tensors outside the embedding: about 100 million trainable numbers, plus 23.6 million in the alpha and beta gates and 2.6 million in the norms and convolutions. That is under half a percent of the model's 26.9 billion weights. You run the model with the codes held fixed, backpropagate into the scales, and nothing about the file format changes. It still loads in stock llama.cpp, because to llama.cpp it is still just an IQ2_XS-family file with different numbers in the scale slots.

What Underdog trained on, and against what objective, is not public. The page says: "This is a preview checkpoint. We'll be publishing our full training soon." The card says "tuned to keep tool calling intact". The obvious reading is tool-calling data (perhaps distilled from the full model, as GSQ does), but that is my inference. What I can say from the bytes is that it is not a fine-tune in the usual sense. No weight crossed to a different grid point. 79% of the weights stayed exactly where ISTA put them, and the model learned to lean on them differently by rescaling blocks of 256.

And the effect is large on the one test where both files were run. On Underdog Bench, ISTA's IQ2_XS scores 76 of 120 and Saluki 88. Same codes in most of the model, half a gigabyte smaller, twelve more tasks passed.

Tool calling: what was tested

Underdog Bench is 120 tasks from the Berkeley Function Calling Leaderboard v4, 24 in each of five kinds. In the page's words: "We picked and froze them on 27 September, before any of these models were tested." Thinking off, temperature 0, each model "in its own runtime".

expand

The 88 against 84 is four tasks out of 120, and Underdog says so itself: "120 tasks is a modest test, and part of that lead is likely run-to-run variation." I agree, and would go further. At a pass rate near 70%, one standard error on 120 pass-or-fail tasks is about five tasks, so 88 and 84 are statistically the same score. "Matches the full model at tool calling" is what the data supports. "Beats" is a headline.

The per-kind split says where Saluki is strong.

expand

Picking a single function is nearly solved: 24 and 23 of 24. The two multi-call rows, 12 and 11 of 24, are where it loses, and the separate parallel test makes the same point. On 100 BFCL v4 parallel tasks with strict scoring, Mia's 2-bit EXL3 build passes 69, Saluki 42 and the full model 35. With a lenient parser that forgives formatting slips, Saluki goes to 55; Underdog reports that on 23 tasks Saluki's reply "didn't contain a call the checker could read". The full model's 35 is low for a reason the page gives: with thinking off it "often makes one call and stops when the task needs several". That 35 is the denominator of the 120% retention row on the launch card.

Two more tests are closer to agent work. On 50 SWE-bench Verified issues Saluki fixed 30 and the full model 33, keeping 27 of the full model's 33 fixes. Bonsai 2 fixed 29. On 150 WebWalkerQA browsing questions Saluki answered 60, and the full model was not run. These are Underdog's own runs, and they are the most informative numbers on the page, because both sides ran in the same harness.

One row in the bar chart is worth a footnote. "Main", Underdog's other build at 7.9 GB, scores 91, the best of all. I couldn't find it among Conway's Hugging Face repositories, and the page explains why it isn't the launch: "Main browses less well, which is one reason we're launching Saluki."

96% of what

The X post says 96%, the card says "96% across 9 benchmarks", and the launch image prints 96.0%. It reproduces exactly. Here are the nine ratios, Saluki over the full model, from the card's own numbers:

The arithmetic mean is 96.05%. Three things about it.

The parallel row contributes 120%, from a full model that stops after one call with thinking off. Drop it and the mean of the other eight is 93.1%. Cap every row at 100%, the reading that "retention" implies, and it is 92.8%. One reply to the launch put it bluntly: "By definition you cannot 'retain' more than 100%".

Six of the nine full-size numbers are public figures from a different harness. Underdog's footnote calls that harness "more lenient", and the page says the IFEval 91.50 comes from Ternary Bonsai 2's model card, which does not say whether it is loose or strict; Saluki's 93.5 is the loose score, and its strict score is 90.9. The six cross-harness rows average 91.5%. The three same-harness rows average 105.2%, and two of those three are tool-calling tests chosen by the people who tuned the model for tool calling.

Math is the honest loss. AIME holds 82 to 85%, MuSR 85%. The card says so up front: "Keeps about 82 to 85% of the full model's competition math score." The odd part is that ISTA's own table puts its untuned IQ2_XS at 96.67 on AIME25 against 100.00 for BF16, a 97% retention, from the very codes Saluki keeps. Different harness again, and ISTA's AIME25 number is on 30 problems, so I wouldn't put the two side by side as a measurement. But it is consistent with the tuning having traded some math for tool calls, which is what you would expect if the scales were fitted to tool-calling data. Nobody has run both files on the same math harness, and that run would settle it.

Bonsai 2 quoted 98.2% retention, and the site's audit found it was a mean over a suite that changed between documents and left out the agentic benchmarks. Saluki's 96% has the opposite shape: the agentic and tool tests are in, and they are what holds it up.

The 7.5x

The thread's third post says Saluki "packs 7.5x more tool-calling capability per gigabyte". The image gives the formula: intelligence density = −log2(1 − score) / size in GB, under the "intelligence density" name PrismML used for Bonsai.

expand

It checks out: −log2(1 − 88/120) is 1.907, divided by 7.89 gives 0.2417, and the full model's 1.737 over 53.8 gives 0.0323, a ratio of 7.49. But 6.82 of that 7.49 is just 53.8 divided by 7.89. The score difference contributes a factor of 1.10. Any 7.9 GB file that ties the full model on this test would be "6.8x more capable per gigabyte". The metric measures size with a small adjustment for score, which is fine as long as you read it that way.

Bits, bytes and what fits

The calculator below puts the builds that come up in this story on one axis. Slide the bits per weight and it computes the weights for this model's 26,895,998,464 parameters, adds the KV cache and the DeltaNet state, and checks the total against a memory budget. Only the 16 full-attention blocks keep a KV cache in Qwen3.8-27B: 4 KV heads of 256 dimensions, keys and values, at 2 bytes, is 65,536 bytes per token at f16, so a 32K context costs 2.15 GB. The 48 DeltaNet blocks keep a fixed recurrent state instead, which by my arithmetic (48 heads of 128 by 128 at f32 per block) is about 0.15 GB at any length. Every score in the table is cited: Underdog Bench from Underdog's page, the reasoning average from ISTA's table. A dash is a number nobody published.

A few things fall out of it. Saluki's 7.90 GB plus a 32K context is 10.2 GB, so the page's "Runs on 16 GB laptop" is comfortable, and an 8 GB card needs layers offloaded to the CPU, whatever "under 8 GB" suggests; one reply on X measured 1.7 tokens a second at a 262K context on a 12 GB card. Unsloth's UD-IQ2_XXS at 7.27 GB is smaller still and in nobody's tool-calling comparison, so "the best model under 8 GB" means best among the builds Underdog tested, and two of the three outside builds it tested are over 8 GB. A stock Q4_K_M-class file is twice Saluki's size; if you have the memory, I'd take that, or ISTA's IQ3_S at 11.8 GB, which ISTA's table puts within 0.17 of BF16 on its reasoning average.

The site's Qwen3.8-27B on a 12 GB card and Qwen3.8-27B variants pieces do the same arithmetic for trellis and EXL3 builds, and the first Bonsai and LittleBit cover what happens below two bits.

Running it, and what "runs in Underdog" means

The card's quickstart is stock llama.cpp:

huggingface-cli download ConwayResearch/Underdog-Saluki-27B-1.0 Underdog-Saluki-27B-1.0-IQ2-mix.gguf --local-dir .
llama-server -m Underdog-Saluki-27B-1.0-IQ2-mix.gguf --jinja -ngl 99 -fa on -c 32768

Every tensor type in the file (IQ1_S through Q8_0) is a mainline ggml type, and the architecture string is qwen35, the same one ISTA's file uses, so anything built on a current llama.cpp should load it: LM Studio, Ollama and the like. That is the real difference from Ternary Bonsai 2, whose PTQ1_0 type mainline llama.cpp rejects, and from Mia's EXL3 build, which needs ExLlamaV3. MLX has no loader for these IQ types, and I wouldn't count on vLLM's GGUF path for a hybrid DeltaNet model full of IQ1 tensors; I tested neither. For fast tool calls the card recommends thinking off with "chat_template_kwargs": {"enable_thinking": false} and temperature 0, and the page recommends a lenient tool-call parser for parallel calls.

"Runs in Underdog" means the Underdog desktop app. Conway publishes the server it uses as ConwayResearch/underdog-llama-server: "llama.cpp's llama-server, built unmodified by Underdog", pinned to commit bd4eeaa047006cb1fe71999fbd11134b5836e167, for Apple-silicon Macs, which the app downloads alongside the model and checks by SHA-256. So the app is a front end on stock llama.cpp, and Saluki is not tied to it. The Husky piece covers the other half of Conway's stack, its own speculative-decoding engine for the smaller Woof models.

Vision is muddled. The card ships two projector files and says "text, plus images with the optional vision add-on", and Underdog replied "Yes it has vision" on X. The Saluki page says "Text in, text out. This release doesn't take images." Both can be true: the GGUF is text-only, and the projector is an add-on the app may not wire up. I didn't test whether the projector, which is the base model's vision tower, works well against retuned 2-bit weights.

The licence chain

Apache 2.0 holds up all the way down. Qwen/Qwen3.8-27B is tagged apache-2.0 on Hugging Face; ISTA's repository is apache-2.0 and says its weights inherit the base model's licence; Saluki ships the Apache 2.0 text and a NOTICE crediting both upstreams, which is what the licence asks of a modified work. The page goes further than it needs to: "You're welcome to use it, even without citing us." The only loose end is cosmetic. The GGUF still introduces itself as ISTA's file, so any tool that prints general.name will show Qwen3.8-27B GSQ-RCO.

What I'd take from it

Saluki is a useful file and an honest one, once you read the card instead of the post. The card names its base, publishes its weak spots (math, letter puzzles, about a fifth of parallel replies with formatting slips), and its benchmark section flags the cross-harness numbers. The marketing layer on top, "96%", "beats Qwen", "7.5x", is each technically reproducible from the card and each leans on something: a 120% row, a four-task gap, a size ratio.

The part I would keep is the technique, because it generalises. A good 2-bit GGUF from a learned quantizer is a fixed lattice of codes plus a thin layer of continuous scales. That thin layer, about 100 million numbers here, is enough to steer the model toward a task by twelve points on a 120-task test, without touching a code and without leaving the stock file format. Anyone with a GSQ-RCO file, a task, and the memory to backpropagate through a 27B model can do the same.

What I'd want before relying on it: the training data and objective Underdog has promised, a run of ISTA's untuned file and Saluki on the same math and coding harness to show what the tuning cost, and an ablation of the ffn_down demotion. Until then, the honest summary is that Saluki matches the full model on single-function tool calls in 7.90 GB, and is clearly worse at math.

How I checked

• Headers. I fetched the first 24 MB of both GGUFs (Saluki's and ISTA's IQ2_XS) with HTTP range requests and parsed them with a small GGUF reader: metadata, 851 tensor records each, type, shape and offset. Bits per weight are tensor bytes times 8 over element count; file sizes come from the Hugging Face API.
• Byte diff. For every tensor I range-fetched the same region from both files at their own offsets: all 353 F32 tensors in full, a 1,000-block window of each of the 352 quantized tensors whose type didn't change, and four windows across the token embedding. Scale positions within each block format follow ggml's block layouts, and I located them from where the differences fell, not from assumption. For the Q8_0 gates I dequantized Saluki's blocks and compared them with ISTA's BF16 values and with an ideal Q8_0 rounding of those values.
• Arithmetic. The 96% mean, the alternative aggregates, the density metric and the KV and state sizes are my arithmetic on the numbers named above. The DeltaNet state size assumes f32 state, which I did not confirm in llama.cpp's code.
• Sources. The model card, NOTICE and LICENSE; ISTA's model card and per-tensor allocation file; GSQ's paper, Figure 1 and Appendix A.2; Underdog's Saluki page (read on 10 October 2026); and the X thread and replies.
• Not checked. I ran no model. Every benchmark number is Underdog's or ISTA's. I couldn't find "Main", and I don't know what Saluki was trained on.