Nar TTS: the two rewards it built and refuses to switch on

2026-08-23 · 7 min · tts · grpo · reinforcement-learning

Why read this

Reads a small TTS repo for its evaluation discipline: two rewards built and held at zero so the judge is never the model it trained against.

Original analysis
A lasting reference

Speech & audioNeeds datacenter GPUsPractitioner tool

How this was scored

Is it new?: 1 of 3: An incremental tweak
Can I trust it?: 1 of 3: Spot-checks a few numbers
Can I run it?: 1 of 3: API-only, gated or restrictive licence
Will I understand it?: 1 of 3: Partial mechanism
Can I act on it?: 1 of 3: General advice
Will it last?: 2 of 3: A reference for a year or more
Does it affect many?: 0 of 3: A niche paper or one-off repo
Only here?: 2 of 3: A teardown or measurement few others did

Score 39 of 100, ranked 535 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Nar TTS combines a user-selected causal language model with Mimi speech tokens. It does quality-gated inference, expressive controls, supervised training and GRPO, without model-specific config files. That description would fit a hundred repositories.

What makes it worth reading is a paragraph in docs/quality.md:

The emotion and non-verbal event reward implementations are ready, but their weights remain zero. Do not enable them until an independently validated classifier has been selected using synthetic Turkish speech. Using the same SER model as both the reward and the success metric encourages reward hacking.

Two rewards, built, wired in, and deliberately held at zero — with the reason written down. That discipline shows up three separate times in this repository, and it is not something most projects with far more resources maintain.


Repo	kadirnar/nar-tts · Python 3.10, CUDA 12.x
Design	any causal LM + Mimi speech tokens · no model-specific config files
Training	pretrain → optional SFT → optional GRPO, FSDP via `accelerate`, LoRA optional
GRPO	8 generations per group across 8 GPUs · six active rewards, two held at zero
Inference	2 candidates, 4 on gate failure · Whisper verification · JSON quality report
Controls	6 emotions × 3 deliveries × 8 non-verbal events, as text markup
Credits	Orpheus TTS, Qwen3, Mimi

Never judge with the model you trained against

one GRPO recipe · 8 generations per group across 8 GPUs · each reward normalized within its groupweights sum to 1.00

intelligibility · weight 0.60 · Qwen3-ASR CER + ground-truth NLL

the same discipline, three times over — never judge with the model you trained against

training reward ASRQwen3-ASR

inference verification ASRWhisper — a different family, on purpose

release evaluation ASR"do not evaluate only with the model used for training"

Sixty per cent of the reward is intelligibility, which is the right allocation for a TTS system and also the least interesting thing here. The interesting thing is the two rows at zero.

Emotion and event rewards are implemented and deliberately switched off, with the reason written down: the project’s own success criteria score emotion accuracy and event F1 using a classifier, and putting that classifier into the reward means optimizing the model’s agreement with the judge rather than the thing the judge stands for. The same instinct shows up twice more — Whisper verifies at inference precisely because the training reward used Qwen3-ASR, and the evaluation section says not to score a release with the model it was trained against.

Building a reward, wiring it in, and then refusing to turn it on until you have a judge you did not train against is a discipline most projects with far more resources do not maintain.

The active reward mix is sensible and unsurprising: 60% intelligibility from Qwen3-ASR CER plus ground-truth NLL, 15% speaker similarity from WavLM-Large and ECAPA, then duration, technical signal quality, prosody and speaker drift. They sum to exactly 1.00, and each is normalized inside its own prompt group before the weighted sum — necessary, because CER, cosine similarity and a clipping diagnostic share no scale.

The two rows at zero are the article. The project's own success criteria include emotion accuracy and event F1, both scored by a classifier. Put that classifier into the reward and the model optimizes its agreement with the judge rather than the thing the judge was standing in for — and the evaluation, which uses the same judge, reports that it worked.

The same instinct appears twice more:

At inference, verification uses Whisper, described as "independent of the Qwen3-ASR training reward". A different ASR family, on purpose.
In the release criteria: "Do not evaluate only with the Qwen3-ASR model used for training. Use an independent ASR family, speaker-drift checks, multi-dimensional quality metrics, and listening tests."

And a sentence I would put on a poster: "technical_quality is a signal diagnostic, not a MOS or naturalness model. An emotion-classifier score alone is not evidence of product quality."

The quality gate is best-of-N with the bill itemized

two candidates · four only when the gate fails · every candidate kept, with a JSON report2.08 candidates for 99.84% success

one candidate clears the gate80%

no gate — take the first candidate

1.00 gen80.0%

adaptive — two, then two more if needed

2.08 gen99.8%

always four

4.00 gen99.8%

bar and first value are generations per request · last column is the chance the request ends with an output that passes

the gate fires

4.0%

of requests need the second pair

against always-four

−48%

same success rate, less compute

what the gate buys

+19.8

points over taking the first candidate

what the gate checks

CERWhisper — a different ASR family from the training reward

speaker similaritydoes it still sound like the reference speaker

durationagainst the reference

clippingsignal diagnostic

silencesignal diagnostic

repetitionthe characteristic autoregressive TTS failure

Adaptive best-of-N reaches exactly the same success rate as always generating four, because the second pair is produced precisely when it is needed — but it only pays for it on the requests that need it. At a gate pass rate of 80% that is 2.08 generations instead of 4, and the scheme degrades gracefully: as the pass rate falls, it converges on always-four rather than failing.

The pass rate is not published and it depends on the checkpoint, the reference clip and the text — which is why the report matters more than the number. Nar writes the winning WAV, every candidate, and a machine-readable JSON report including a real-time factor, so the pass rate is something you measure on your own data rather than a parameter you inherit. That is the difference between best-of-N as a quality feature and best-of-N as a cost you cannot account for.

Default inference generates two candidates, verifies them, and expands to four only when the gate fails. The gate checks CER, speaker similarity, duration, clipping, silence and repetition — with the last one being the characteristic failure of autoregressive TTS, and the one a listener notices immediately.

Adaptive best-of-N reaches exactly the same success rate as always generating four, because the second pair is produced precisely when it is needed. It just only pays for it on the requests that need it: at a gate pass rate of 80%, that is 2.08 generations instead of 4, and it degrades gracefully rather than failing as the pass rate drops.

The part that makes this a system rather than a trick is what it writes out: the winning WAV, every candidate, and a machine-readable JSON report including a real-time factor. The docs are explicit that best-of-N and verification cost extra compute and that real_time_factor is how you compare speed against quality on your own data. A best-of-N scheme whose cost you can measure is a feature; one whose cost is invisible is a surprise on the invoice.

There is also a content-addressed Mimi token cache for repeated reference audio, true batched generation with a KV cache, and sentence splitting for long text with acoustic context carried from the previous chunk plus crossfading. That last one is the detail that separates a demo from something you would run on a paragraph.

Controls that do not require retraining

three independent controls, rendered as text — no new tokens, no re-encodingthe shipped checkpoint has not learned this markup

emotion

delivery

event

intensity0.90

after word1

what the model sees

<nar_control emotion=sadness intensity=0.900 delivery=crying_speech>
I <nar_event type=sob after_word=1 duration=short count=1> thought of you today.

speech_laugh means the text is spoken with laughter; laugh is a separate laughter event. Likewise crying_speech and sob are annotated separately — a distinction most expressive TTS interfaces collapse into one label.

Controls live in the text stream rather than in new special tokens, and that is not a stylistic preference. The repository has a section called changes that require retraining, and adding control tokens is on it, next to changing the codec or the speech-token layout — because any of those means re-encoding every piece of speech data and training the model again. Markup that the existing tokenizer already handles keeps expressive control on the near side of that line.

And then the honest part, which is in the same docs: the shipped checkpoint has not learned this markup and cannot produce crying speech or laughter on its own. The interface exists; the capability needs labelled expressive SFT data. Publishing a control surface and stating plainly that the current weights ignore it is the opposite of how this usually goes.

Nar exposes three independent controls — six emotions, three deliveries, eight non-verbal events — and renders them as text markup that does not modify the tokenizer.

That is a deliberate architectural choice, and the repository explains the cost of the alternative in a section called changes that require retraining: if the codec, the speech-token layout, or the special control tokens change, all speech data must be re-encoded and the model retrained. Alternative codecs, a new decoder and true frame-level streaming are all listed there too, as separate model generations rather than upgrades. Keeping expressive control in the text stream keeps it on the near side of that line.

The taxonomy is more careful than most, too: speech_laugh means the text is spoken with laughter while laugh is a separate laughter event, and crying_speech and sob are annotated separately. Most expressive TTS interfaces collapse those into one label and then wonder why the control is unreliable.

And then, in the same document: "The current checkpoint has not learned this markup, so it cannot produce crying speech or laughter on its own. These capabilities require an expressive SFT checkpoint." The interface is shipped; the capability is a labelled-data problem, and the repo says so rather than letting the API imply otherwise.

The data loop

The pipeline in docs/quality.md is worth drawing because it closes.

raw manifest → … → hard cases → the next roundnar-tts distill

hard cases / distillnar-tts distill

The step worth stealing. Only the winning best-of-N samples that passed every threshold are added to the new SFT manifest, tagged hard_case=true so they can be sampled more frequently during GRPO. The inference-time quality gate was already computing this for every request — so the gate that protects production is also the hard-example miner, for free.

The arc at the bottom is what makes this a loop rather than a pipeline, and the step it comes from is the one worth taking to another project. The inference-time quality gate is already computing, for every single request, whether an output cleared every threshold. So the gate that protects production is also, at zero additional cost, a miner for exactly the examples the next training round should see more of.

Only winners that passed everything are added, tagged hard_case=true, and those rows can then be oversampled during GRPO. Most projects build a serving quality gate and a training data pipeline as two independent systems, and pay for both.

Where the thin parts are

There are no numbers. The success criteria are well specified — Turkish, English and Japanese CER and WER, speaker similarity and long-form drift, p50/p95 RTF, VRAM, time to first chunk, clipping/silence/repetition/truncation rates, blinded human A/B — and none of them are reported. This is a repository that documents how it would evaluate a release without having published one.

The GRPO weights are asserted, not ablated. 0.60 / 0.15 / 0.08 / 0.07 / 0.05 / 0.05 is a plausible allocation and nothing shows what happens if intelligibility drops to 0.4, or what the speaker-drift term is worth on long-form output. For a repository this careful about not fooling itself, the weights are the one place that reads as taste rather than measurement.

"Any causal LM" is a strong claim with one worked example. The design goal is no model-specific config files — you run inspect-tokenizer, copy text_eos_token_id and pad_token_id into three YAMLs, and go. Qwen3 is credited. Whether a Mimi speech-token head grafts equally well onto a differently-shaped model is the claim the architecture rests on, and it is untested in public.

The Turkish focus is a feature and a constraint. Examples are Turkish, normalization covers Turkish and English, and the reward-hacking warning specifically says to validate the emotion classifier on synthetic Turkish speech. Good TTS tooling for languages outside the usual five is genuinely undersupplied. It also means the evaluation set the whole quality system is designed around does not exist publicly yet.

Why a repository with no benchmarks is worth an article

Nar TTS has no results table, no model weights, and no demo I can point at. What it has is a set of decisions written down in the order they were made, each with its reason attached — and several of those decisions are ones larger projects get wrong.

Do not use the same model as reward and metric. Do not evaluate a release with the ASR you trained against. Do not claim a control surface works when the checkpoint has not learned it. Do not put controls in special tokens if you cannot afford to re-encode your corpus. Do not report a signal diagnostic as a naturalness score.

None of those are novel. All of them are the kind of thing that gets skipped under deadline, and the resulting model looks great on the metric it was trained to satisfy. A repository that writes them down as rules — and then holds two finished rewards at zero to obey one — is worth more attention than its star count suggests.