# Underdog Saluki 27B: ISTA's 2-bit file with its scales retuned
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/underdog-saluki-27b
> date: 2026-10-10
> tags: quantization, qwen, tool-calling, llama-cpp, gguf, benchmarks, on-device
Underdog's launch post said three things in one breath: Qwen3.8-27B, "almost 7x smaller", keeps "96% of its benchmark performance" and "beats Qwen at tool calling". A 27B model in under 8 GB is about 2.3 bits per weight. At that width every quantizer I know of loses something real, and tool calling is usually one of the first things to go, because a single wrong token in a JSON argument fails the whole call. So when a 2-bit build claims to *beat* the full model there, either somebody has a new quantizer, or the comparison is doing some of the work.
It turned out to be neither, exactly. The model card says plainly that Saluki is built on someone else's 2-bit file. The interesting question became what Underdog did to that file, and that is something you can read straight out of the bytes.
## What is in the file
The repository is [`ConwayResearch/Underdog-Saluki-27B-1.0`](https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0) (ConwayResearch is Underdog's organisation on Hugging Face; its `NOTICE` reads "Underdog (ConwayResearch)"). It holds one GGUF, `Underdog-Saluki-27B-1.0-IQ2-mix.gguf`, at 7,898,369,152 bytes, plus two optional vision projectors and an Apache 2.0 `LICENSE` and `NOTICE`.
I pulled the first 24 MB of the file with an HTTP range request and parsed the GGUF header. It lists 851 tensors and 26,895,998,464 weights: the text decoder, the token embedding and the LM head, with no vision tower and no multi-token-prediction head. The tensor data adds up to 7,887,374,336 bytes, which is 2.346 bits per weight; with the 11 MB header (most of it the tokenizer) the whole file is 2.349. Against ISTA's BF16 reference at 53.8 GB that is 6.8x smaller, so "almost 7x" is fair.
Then the metadata. `general.name` is `Qwen3.8-27B GSQ-RCO`. `general.quantized_by` is `ISTA DASLab`. The description field reads "GSQ-RCO non-uniform mixed-precision GGUF (Deep Algorithms and Systems Lab, ISTA)". Underdog didn't hide this. The card's front matter lists `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` as a base model, and the `NOTICE` file says it outright: "Underdog Saluki 27B 1 is built on this release", naming the file `Qwen3.8-27B-GSQ-RCO-IQ2_XS.gguf`. The launch post and the X thread never mention it, which is a shame, because the base is the most interesting part of the story.
Only three fields in the metadata differ from ISTA's file: `general.file_type` went from 19 to 20, `general.quantization_version` moved, and the importance-matrix path became `/root/work/imatrix-qwen3.8-27b.gguf`, which is the same file name ISTA ships in its repository. Those are the fields `llama-quantize` writes. So somebody ran ISTA's file back through the stock quantizer. That much is plumbing. What came out of it is not.
## What ISTA built, and why it is already good
ISTA-DASLab's release ([`ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF)) uses two of their own methods. The file formats are llama.cpp's stock ones, so the cleverness is in which values go into them.
A llama.cpp IQ2 tensor is stored in super-blocks of 256 weights. Each super-block holds one fp16 scale `d`, a handful of small sub-block scales, and codes that index a fixed grid of points (for `IQ2_XXS`, points on the E8 lattice, eight weights at a time, with sign bits). The 66 bytes of an `IQ2_XXS` block are the 2-byte scale plus 64 bytes of codes, which is where its 2.0625 bits per weight comes from. Every format in the file follows the same pattern: a few bytes of continuous scale, and the rest discrete choices.
The usual way to fill those codes is rounding, guided by an importance matrix: GPTQ-style error feedback, or llama.cpp's own search for the grid point nearest each group of weights. [GSQ](https://arxiv.org/abs/2604.18556) (Gumbel-Softmax Quantization) treats the code choice as something to learn instead. Each coordinate gets logits over nearby grid points, a Gumbel-Softmax relaxation makes the choice differentiable, and the logits and the per-group scales are trained together, block by block, against the full-precision model's outputs. At higher bit widths it only learns a shift of up to two grid steps away from the GPTQ starting point, which keeps the number of logits per weight at five.
[RCO](https://arxiv.org/abs/2605.00649) then decides how many bits each tensor gets. With every tensor quantized at every candidate type, it searches for one type per tensor so that the file hits a size budget exactly, optimising directly on the task loss. ISTA ships the result as a text file per build. For the 2.50 bits per weight build, the histogram runs from 31 tensors at `IQ1_M` and 36 at `IQ1_S` up to 11 at `IQ4_XS` and 4 at `Q4_K`. The token embedding gets `IQ1_M` (1.75 bits; presumably because a lookup only ever reads one row, so its errors stay local), the LM head gets `IQ4_XS`, and the Gated DeltaNet `ssm_alpha` and `ssm_beta` gates stay in BF16.
ISTA's own table shows why Underdog picked this file. At 8.4 GB, `IQ2_XS` averages 86.03 across AIME25, GPQA-Diamond and LiveCodeBench v6 against 91.87 for BF16, while Unsloth's `UD-IQ2_S` at the same size averages 78.31. On AIME25 alone the gap is ten points: 96.67 against 86.67.
The site has looked at this family before. [Mach-2 Additive](/articles/mach-2-additive) used a GSQ-RCO build as its "traditional quant" foil, and [runtime dynamic compression](/articles/runtime-dynamic-compression) compared against GSQ-RCO on the same dense 27B. What nobody had done was take one of these files and keep working on it.
## Two files, byte by byte
Both files are public, so I compared them tensor by tensor. Same 851 names, same shapes. 705 tensors kept their type; 146 changed. The changes fall into three groups:
| change | tensors | bytes saved |
|---|---|---|
| FFN projections re-quantized to `IQ2_XXS` | 49 | 373,227,520 |
| LM head `IQ4_XS` → `IQ3_S` | 1 | 129,126,400 |
| DeltaNet `ssm_alpha` / `ssm_beta` BF16 → `Q8_0` | 96 | 22,118,400 |
Together that is 524,472,320 bytes, which is what takes ISTA's 8.42 GB to Saluki's 7.90 GB and under the 8 GB line on the launch card.
The 705 tensors that kept their type are where it gets interesting. If Underdog had simply re-run `llama-quantize`, those would be bit-identical: when the target type equals the source type, the quantizer copies the bytes through. Some are. The token embedding, all 278 MB of it, matches byte for byte at every offset I sampled, and so does `output_norm`. But the first tensor I tried after those, block 0's `attn_qkv`, did not match.
So I looked at where in each 256-weight block the bytes differ. For `IQ3_S`, `IQ2_XS`, `IQ2_XXS`, `IQ1_S` and `IQ4_XS` it is bytes 0 and 1, and nothing else. Those two bytes are the fp16 super-block scale `d`. For `Q2_K` it is bytes 80 to 83, which are `d` and `dmin`. For `IQ1_M`, whose scale is split across the high nibbles of its sub-scale field, it is bytes 49, 51 and 53. In every format, the codes are untouched and the scale moved.
```python
# for one tensor present in both files at the same type
a = fetch(SALUKI, off_s, nblocks * bb).reshape(-1, bb) # uint8, one row per block
b = fetch(ISTA, off_i, nblocks * bb).reshape(-1, bb)
print(np.nonzero((a != b).any(0))[0]) # IQ2_XS -> [0 1]; Q2_K -> [80 81 82 83]
d_s = a[:, 0:2].copy().view(np.float16).astype(np.float64)
d_i = b[:, 0:2].copy().view(np.float16).astype(np.float64)
print(np.median(np.abs(d_s / d_i - 1))) # ~0.01
```
I ran that over a 1,000-block window of every one of the 352 quantized tensors whose type didn't change. Not one code byte differs anywhere. About 98% of block scales moved (the median tensor), by a median of 1.0%, with the 99th percentile around 4%. The changes are centred on zero and look like the residue of an optimiser, not a rescale. In `Q2_K` blocks, `d` and `dmin` moved by the same factor, with a correlation of 0.9997 between their ratios, so one multiplier per block was learned and applied to both. The only tensor whose scales didn't move is the token embedding.
The continuous tensors moved too. Every `attn_norm` and `post_attention_norm` in all 64 blocks differs from ISTA's, by about 3e-4 relative. The DeltaNet convolution weights differ by about 0.8%. The 96 `ssm_alpha` and `ssm_beta` tensors are now `Q8_0`, and they sit 1.5 to 2% away from ISTA's BF16 values, against 0.5 to 0.6% for a clean `Q8_0` rounding of the same numbers. They were changed in full precision first and quantized afterwards.
This is training. The codes, 79% of the model's weights by count (the 19.96 billion in same-type quantized tensors plus the 1.27 billion in the untouched embedding), are frozen exactly as ISTA's search left them. Everything continuous around them was optimised.
Switch the grid between the two files and the re-typing has an obvious shape. None of it touches the attention or DeltaNet mixers. Almost all of it is FFN down: 41 of 64 `ffn_down` tensors went to `IQ2_XXS`, dropping that role from 2.545 to 2.127 bits per weight, plus six `ffn_up` and two `ffn_gate`. What surprised me is where it lands. RCO had spent some of its most expensive types on the late `ffn_down` projections, `IQ3_S` and `IQ4_XS` in blocks 50 to 57, and Underdog flattened most of those to the cheapest 2-bit type in the file. A gradient-based search said those tensors were sensitive. Underdog overrode it, presumably to hit the size target with one blunt rule, and then trained to recover the damage. There is no ablation that separates the two, so I can't tell you how much the demotion cost or how much the tuning bought back.
## The recipe is in GSQ's own appendix
Freezing the codes and training only the scales is not a trick Underdog invented. It is the last stage of GSQ. Appendix A.2 of the GSQ paper, "Effect of end-to-end scale-only fine-tuning", describes it: "the discrete assignments found by block-wise optimization are kept fixed, and only the per-group scales are updated using the distillation objective". On 2-bit Llama-3.1-8B-Instruct it lifts the five-task zero-shot average from 65.54 to 68.55, plus 3.01 points, from a single epoch over 4,096 sequences.
It is cheap for a reason. There are 99,993,600 super-block scales in Saluki's quantized tensors outside the embedding: about 100 million trainable numbers, plus 23.6 million in the alpha and beta gates and 2.6 million in the norms and convolutions. That is under half a percent of the model's 26.9 billion weights. You run the model with the codes held fixed, backpropagate into the scales, and nothing about the file format changes. It still loads in stock llama.cpp, because to llama.cpp it is still just an `IQ2_XS`-family file with different numbers in the scale slots.
What Underdog trained on, and against what objective, is not public. The page says: "This is a preview checkpoint. We'll be publishing our full training soon." The card says "tuned to keep tool calling intact". The obvious reading is tool-calling data (perhaps distilled from the full model, as GSQ does), but that is my inference. What I can say from the bytes is that it is not a fine-tune in the usual sense. No weight crossed to a different grid point. 79% of the weights stayed exactly where ISTA put them, and the model learned to lean on them differently by rescaling blocks of 256.
And the effect is large on the one test where both files were run. On Underdog Bench, ISTA's `IQ2_XS` scores 76 of 120 and Saluki 88. Same codes in most of the model, half a gigabyte smaller, twelve more tasks passed.
## Tool calling: what was tested
Underdog Bench is 120 tasks from the [Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) v4, 24 in each of five kinds. In the page's words: "We picked and froze them on 27 September, before any of these models were tested." Thinking off, temperature 0, each model "in its own runtime".
The 88 against 84 is four tasks out of 120, and Underdog says so itself: "120 tasks is a modest test, and part of that lead is likely run-to-run variation." I agree, and would go further. At a pass rate near 70%, one standard error on 120 pass-or-fail tasks is about five tasks, so 88 and 84 are statistically the same score. "Matches the full model at tool calling" is what the data supports. "Beats" is a headline.
The per-kind split says where Saluki is strong.
Picking a single function is nearly solved: 24 and 23 of 24. The two multi-call rows, 12 and 11 of 24, are where it loses, and the separate parallel test makes the same point. On 100 BFCL v4 parallel tasks with strict scoring, Mia's 2-bit EXL3 build passes 69, Saluki 42 and the full model 35. With a lenient parser that forgives formatting slips, Saluki goes to 55; Underdog reports that on 23 tasks Saluki's reply "didn't contain a call the checker could read". The full model's 35 is low for a reason the page gives: with thinking off it "often makes one call and stops when the task needs several". That 35 is the denominator of the 120% retention row on the launch card.
Two more tests are closer to agent work. On 50 SWE-bench Verified issues Saluki fixed 30 and the full model 33, keeping 27 of the full model's 33 fixes. Bonsai 2 fixed 29. On 150 WebWalkerQA browsing questions Saluki answered 60, and the full model was not run. These are Underdog's own runs, and they are the most informative numbers on the page, because both sides ran in the same harness.
One row in the bar chart is worth a footnote. "Main", Underdog's other build at 7.9 GB, scores 91, the best of all. I couldn't find it among Conway's Hugging Face repositories, and the page explains why it isn't the launch: "Main browses less well, which is one reason we're launching Saluki."
## 96% of what
The X post says 96%, the card says "**96%** across 9 benchmarks", and the launch image prints 96.0%. It reproduces exactly. Here are the nine ratios, Saluki over the full model, from the card's own numbers:
| benchmark | Saluki | full size | ratio | full-size source |
|---|---|---|---|---|
| Underdog Bench | 88 | 84 | 104.8% | Underdog's run |
| BFCL v4 parallel | 42 | 35 | 120.0% | Underdog's run |
| SWE-bench Verified (50) | 30 | 33 | 90.9% | Underdog's run |
| IFEval (loose) | 93.5 | 91.5 | 102.2% | public |
| IFBench (loose) | 72.7 | 71.0 | 102.4% | public |
| MBPP+ | 78.0 | 83.9 | 93.0% | public |
| MuSR | 67.5 | 79.6 | 84.8% | public |
| AIME 2025 | 79.2 | 96.7 | 81.9% | public |
| AIME 2026 | 80.0 | 94.6 | 84.6% | public |
The arithmetic mean is 96.05%. Three things about it.
The parallel row contributes 120%, from a full model that stops after one call with thinking off. Drop it and the mean of the other eight is 93.1%. Cap every row at 100%, the reading that "retention" implies, and it is 92.8%. One reply to the launch put it bluntly: "By definition you cannot 'retain' more than 100%".
Six of the nine full-size numbers are public figures from a different harness. Underdog's footnote calls that harness "more lenient", and the page says the IFEval 91.50 comes from Ternary Bonsai 2's model card, which does not say whether it is loose or strict; Saluki's 93.5 is the loose score, and its strict score is 90.9. The six cross-harness rows average 91.5%. The three same-harness rows average 105.2%, and two of those three are tool-calling tests chosen by the people who tuned the model for tool calling.
Math is the honest loss. AIME holds 82 to 85%, MuSR 85%. The card says so up front: "Keeps about 82 to 85% of the full model's competition math score." The odd part is that ISTA's own table puts its untuned `IQ2_XS` at 96.67 on AIME25 against 100.00 for BF16, a 97% retention, from the very codes Saluki keeps. Different harness again, and ISTA's AIME25 number is on 30 problems, so I wouldn't put the two side by side as a measurement. But it is consistent with the tuning having traded some math for tool calls, which is what you would expect if the scales were fitted to tool-calling data. Nobody has run both files on the same math harness, and that run would settle it.
[Bonsai 2](/articles/bonsai-2-27b) quoted 98.2% retention, and the site's audit found it was a mean over a suite that changed between documents and left out the agentic benchmarks. Saluki's 96% has the opposite shape: the agentic and tool tests are in, and they are what holds it up.
## The 7.5x
The thread's third post says Saluki "packs 7.5x more tool-calling capability per gigabyte". The image gives the formula: intelligence density = −log2(1 − score) / size in GB, under the "intelligence density" name PrismML used for Bonsai.
It checks out: −log2(1 − 88/120) is 1.907, divided by 7.89 gives 0.2417, and the full model's 1.737 over 53.8 gives 0.0323, a ratio of 7.49. But 6.82 of that 7.49 is just 53.8 divided by 7.89. The score difference contributes a factor of 1.10. Any 7.9 GB file that ties the full model on this test would be "6.8x more capable per gigabyte". The metric measures size with a small adjustment for score, which is fine as long as you read it that way.
## Bits, bytes and what fits
The calculator below puts the builds that come up in this story on one axis. Slide the bits per weight and it computes the weights for this model's 26,895,998,464 parameters, adds the KV cache and the DeltaNet state, and checks the total against a memory budget. Only the 16 full-attention blocks keep a KV cache in Qwen3.8-27B: 4 KV heads of 256 dimensions, keys and values, at 2 bytes, is 65,536 bytes per token at f16, so a 32K context costs 2.15 GB. The 48 DeltaNet blocks keep a fixed recurrent state instead, which by my arithmetic (48 heads of 128 by 128 at f32 per block) is about 0.15 GB at any length. Every score in the table is cited: Underdog Bench from Underdog's page, the reasoning average from ISTA's table. A dash is a number nobody published.
A few things fall out of it. Saluki's 7.90 GB plus a 32K context is 10.2 GB, so the page's "Runs on 16 GB laptop" is comfortable, and an 8 GB card needs layers offloaded to the CPU, whatever "under 8 GB" suggests; one reply on X measured 1.7 tokens a second at a 262K context on a 12 GB card. Unsloth's `UD-IQ2_XXS` at 7.27 GB is smaller still and in nobody's tool-calling comparison, so "the best model under 8 GB" means best among the builds Underdog tested, and two of the three outside builds it tested are over 8 GB. A stock `Q4_K_M`-class file is twice Saluki's size; if you have the memory, I'd take that, or ISTA's `IQ3_S` at 11.8 GB, which ISTA's table puts within 0.17 of BF16 on its reasoning average.
The site's [Qwen3.8-27B on a 12 GB card](/articles/qwen3-8-27b-on-12gb) and [Qwen3.8-27B variants](/articles/qwen3-8-27b-variants) pieces do the same arithmetic for trellis and EXL3 builds, and [the first Bonsai](/articles/bonsai-27b) and [LittleBit](/articles/littlebit-0-1-bit) cover what happens below two bits.
## Running it, and what "runs in Underdog" means
The card's quickstart is stock llama.cpp:
```bash
huggingface-cli download ConwayResearch/Underdog-Saluki-27B-1.0 Underdog-Saluki-27B-1.0-IQ2-mix.gguf --local-dir .
llama-server -m Underdog-Saluki-27B-1.0-IQ2-mix.gguf --jinja -ngl 99 -fa on -c 32768
```
Every tensor type in the file (`IQ1_S` through `Q8_0`) is a mainline ggml type, and the architecture string is `qwen35`, the same one ISTA's file uses, so anything built on a current llama.cpp should load it: LM Studio, Ollama and the like. That is the real difference from [Ternary Bonsai 2](/articles/bonsai-2-27b), whose `PTQ1_0` type mainline llama.cpp rejects, and from Mia's EXL3 build, which needs ExLlamaV3. MLX has no loader for these IQ types, and I wouldn't count on vLLM's GGUF path for a hybrid DeltaNet model full of `IQ1` tensors; I tested neither. For fast tool calls the card recommends thinking off with `"chat_template_kwargs": {"enable_thinking": false}` and temperature 0, and the page recommends a lenient tool-call parser for parallel calls.
"Runs in Underdog" means the Underdog desktop app. Conway publishes the server it uses as [`ConwayResearch/underdog-llama-server`](https://huggingface.co/ConwayResearch/underdog-llama-server): "llama.cpp's `llama-server`, built unmodified by Underdog", pinned to commit `bd4eeaa047006cb1fe71999fbd11134b5836e167`, for Apple-silicon Macs, which the app downloads alongside the model and checks by SHA-256. So the app is a front end on stock llama.cpp, and Saluki is not tied to it. The [Husky](/articles/husky-msi) piece covers the other half of Conway's stack, its own speculative-decoding engine for the smaller Woof models.
Vision is muddled. The card ships two projector files and says "text, plus images with the optional vision add-on", and Underdog replied "Yes it has vision" on X. The Saluki page says "Text in, text out. This release doesn't take images." Both can be true: the GGUF is text-only, and the projector is an add-on the app may not wire up. I didn't test whether the projector, which is the base model's vision tower, works well against retuned 2-bit weights.
## The licence chain
Apache 2.0 holds up all the way down. `Qwen/Qwen3.8-27B` is tagged `apache-2.0` on Hugging Face; ISTA's repository is `apache-2.0` and says its weights inherit the base model's licence; Saluki ships the Apache 2.0 text and a `NOTICE` crediting both upstreams, which is what the licence asks of a modified work. The page goes further than it needs to: "You're welcome to use it, even without citing us." The only loose end is cosmetic. The GGUF still introduces itself as ISTA's file, so any tool that prints `general.name` will show `Qwen3.8-27B GSQ-RCO`.
## What I'd take from it
Saluki is a useful file and an honest one, once you read the card instead of the post. The card names its base, publishes its weak spots (math, letter puzzles, about a fifth of parallel replies with formatting slips), and its benchmark section flags the cross-harness numbers. The marketing layer on top, "96%", "beats Qwen", "7.5x", is each technically reproducible from the card and each leans on something: a 120% row, a four-task gap, a size ratio.
The part I would keep is the technique, because it generalises. A good 2-bit GGUF from a learned quantizer is a fixed lattice of codes plus a thin layer of continuous scales. That thin layer, about 100 million numbers here, is enough to steer the model toward a task by twelve points on a 120-task test, without touching a code and without leaving the stock file format. Anyone with a GSQ-RCO file, a task, and the memory to backpropagate through a 27B model can do the same.
What I'd want before relying on it: the training data and objective Underdog has promised, a run of ISTA's untuned file and Saluki on the same math and coding harness to show what the tuning cost, and an ablation of the `ffn_down` demotion. Until then, the honest summary is that Saluki matches the full model on single-function tool calls in 7.90 GB, and is clearly worse at math.
## How I checked
- **Headers.** I fetched the first 24 MB of both GGUFs (Saluki's and ISTA's `IQ2_XS`) with HTTP range requests and parsed them with a small GGUF reader: metadata, 851 tensor records each, type, shape and offset. Bits per weight are tensor bytes times 8 over element count; file sizes come from the Hugging Face API.
- **Byte diff.** For every tensor I range-fetched the same region from both files at their own offsets: all 353 F32 tensors in full, a 1,000-block window of each of the 352 quantized tensors whose type didn't change, and four windows across the token embedding. Scale positions within each block format follow ggml's block layouts, and I located them from where the differences fell, not from assumption. For the `Q8_0` gates I dequantized Saluki's blocks and compared them with ISTA's BF16 values and with an ideal `Q8_0` rounding of those values.
- **Arithmetic.** The 96% mean, the alternative aggregates, the density metric and the KV and state sizes are my arithmetic on the numbers named above. The DeltaNet state size assumes f32 state, which I did not confirm in llama.cpp's code.
- **Sources.** The model card, `NOTICE` and `LICENSE`; ISTA's model card and per-tensor allocation file; GSQ's paper, Figure 1 and Appendix A.2; Underdog's Saluki page (read on 10 October 2026); and the X thread and replies.
- **Not checked.** I ran no model. Every benchmark number is Underdog's or ISTA's. I couldn't find "Main", and I don't know what Saluki was trained on.