# JOSIE-2: a 4M-token fine-tune, and a labeling bug in its own benchmark table

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/josie-2
> date: 2026-08-03
> tags: open-weights, llm, fine-tuning, explainer
[JOSIE-2](https://huggingface.co/collections/Goekdeniz-Guelmez/josie-2) is Gökdeniz Gülmez's third-generation personality-tuned model family: three sizes — 2B, 4B, 9B — each fine-tuned from the matching Qwen3.5 base, MIT-licensed, trained entirely on Apple Silicon. The release note makes a specific, checkable claim: *"JOSIE-2-2B-OSS consistently outperforms its 4B base model. JOSIE-2-4B-OSS consistently outperforms its 9B base model."* A 2B model beating a 4B, and a 4B beating a 9B, would be a genuinely notable result. I went and checked it against the model cards and `config.json` directly, because that's the kind of claim worth verifying before repeating.

## What the cards actually say

Each `JOSIE-2-<size>-OSS` repo carries a `base_model` field in its frontmatter and a `model_name` field in `config.json`. Both agree, on all three repos: **2B is fine-tuned from Qwen/Qwen3.5-2B, 4B from Qwen/Qwen3.5-4B, 9B from Qwen/Qwen3.5-9B.** Same size, every time. There is no size-up training anywhere in the released weights — the release note's "outperforms its 4B/9B base" framing describes a comparison that, per the models' own configs, never happened.

<Figure
  src="/articles/josie-2/fig1.png"
  alt="JOSIE-2-2B-OSS benchmark table from the model's own Hugging Face card. The baseline row is labeled 'Qwen3.5-2B /r (base)' but its adjacent Params column reads '4B' — the same size shown as 4B for every other row on the same table, including the actual 2B model."
  caption="The 2B-OSS card's own benchmark chart (Hugging Face, Goekdeniz-Guelmez/JOSIE-2-2B-OSS): the baseline is named Qwen3.5-2B, but its Params column still reads 4B — a template value left over from a shared table, not corrected per model."
/>

That mismatched "4B" badge on a row that says "Qwen3.5-2B" is the same bug in miniature. The bigger version of it lives in the part of each card that Hugging Face's benchmark widget actually reads — the `model-index` YAML. There, **every one of the three cards labels its baseline row `Qwen/Qwen3.5-4B (base)`, including the 2B and 9B cards**, even though the numbers next to that label differ card to card (82.5/49.1 on the 2B card, 83.4/48.9 on the 4B card, 92.6/69.5 on the 9B card) in a way that only makes sense if each card's numbers really are its own base model's, mislabeled:

<LabelChecker />

Toggle between what's published and what `config.json` verifies, and the numbers don't move — only the caption does. That's what makes this read as a copy-paste templating bug rather than a fabricated result: the underlying scores look real and internally consistent per-card, but the machine-readable label attached to them is wrong on two of three cards. The 9B card has a second, unrelated gap: its own reasoning-mode benchmark row is simply blank, published as "comming soon" in the card's own chart — the one number that would most directly support "the 9B model in its best mode," and it doesn't exist yet.

<Callout type="note">
I want to be fair to the release here: this reads as a labeling artifact, not a fabricated claim. The scores are plausible and self-consistent within each card. The problem is narrower and more mundane — a shared table template where the label field wasn't updated per model, which happens to be exactly the kind of error you'd only catch by checking `config.json` against what the benchmark table says, which almost nobody does.
</Callout>

## What's actually worth taking seriously

None of this makes the underlying work uninteresting. The whole JOSIE-2 family — three sizes — was fine-tuned on **the same dataset**: roughly 3,500 samples, about 4 million tokens total, generated by a pipeline that itself leaned on larger models (Gemma 4 31B, Qwen3.5-9B-Base, GPT-5.4, and an unreleased JOSIE-2-35B-A3B-RTG model) to synthesize training data far more capable than the 4M-token set alone would suggest. That's a striking ratio: a few thousand curated examples, reused across three model sizes, apparently doing real work — the ARC-C and TruthfulQA gains over each model's *actual, same-size* base are consistent and positive across all three sizes, even once you correct the label.

The more genuinely interesting finding in the cards is emergent, not benchmarked: JOSIE-2's reasoning traces sometimes "roast" or insult the user mid-thought, and the cards are specific that *"no reward was introduced to enforce a uniformly polite, corporate, or sanitized internal monologue"* — the behavior wasn't trained in, it showed up during RL because nothing trained it out. It's confined to the hidden reasoning trace, not the user-facing reply, and the cards are candid that they can't yet separate "reasoning-first supervision improves the policy" from "the training data's honesty framing drives the gain" — an open question, stated as one.

## The take

Check the claim, not just the headline: a 2B model plausibly outperforming its own 2B base and a 9B model plausibly outperforming its own 9B base is still a real, useful result from a tiny dataset — it just isn't the size-up story the release note tells, and the cards' own tables currently say something they don't mean to say. That's worth fixing on Gülmez's end, and worth checking on everyone else's before the "2B beats a 4B" framing gets repeated as fact.

---

*Sources: the [JOSIE-2 collection](https://huggingface.co/collections/Goekdeniz-Guelmez/josie-2) and individual model cards (`Goekdeniz-Guelmez/JOSIE-2-2B-OSS`, `-4B-OSS`, `-9B-OSS`), their `config.json` files, and each repo's `benchmarks.png`, cross-checked directly for this piece. The figure is the 2B card's own chart, unedited aside from flattening onto a white background; the interactive reproduces the model-index labels and config-verified bases as published.*