# Audex: audio, speech, and text through one decoder — without the text tax

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nemotron-audex
> date: 2026-07-20
> tags: audio, llm, multimodal, speech, explainer
There's a tax you pay when you make a text LLM multimodal. Bolt on a speech encoder, fine-tune for
audio tasks, and the model's text scores — reasoning, knowledge, instruction-following — tend to
sag. The audio ability arrives; some of the intelligence leaves. NVIDIA's
[Audex](https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B) (Nemotron-Labs-Audex-30B-A3B,
[arXiv 2607.05196](https://arxiv.org/abs/2607.05196)) is a unified audio-text LLM whose whole point
is that it *doesn't* pay that tax. It does ASR, speech translation, audio understanding,
text-to-speech, text-to-audio, and direct speech-to-speech — and keeps the frontier text scores of
the model it was built on.

The design is almost aggressively simple, which is the interesting part.

## One decoder, one vocabulary

Most audio LLMs treat audio as a side channel: an encoder produces features, an adapter head or a
separate module consumes or emits them. Audex refuses the split. It is a **single Transformer
decoder** built on **Nemotron-Cascade-2-30B-A3B** — a hybrid Mamba–Transformer mixture-of-experts,
30B total parameters with ~3B active, a 1M-token context. Audio *inputs* are turned into continuous
embeddings by an audio encoder plus MLP adapters and dropped straight into the **text embedding
space**. Audio *outputs* are discrete tokens drawn from an **extended vocabulary** (reported at
205,312 entries) that the model predicts in exactly the same autoregressive stream as text.

<Figure
  src="/articles/nemotron-audex/fig1.png"
  alt="Audex architecture: speech or general audio enters an audio encoder and MLP adapters into the Nemotron-Cascade-2-30B-A3B backbone alongside text tokens; the backbone emits text tokens directly, speech tokens into a speech decoder, and audio tokens into an audio decoder."
  caption="Audex reads audio as continuous embeddings projected into the text space, and writes text, speech, and general-audio tokens from one extended vocabulary; the discrete speech/audio tokens are detokenized by dedicated decoders (paper, Figure 1)."
/>

The upshot is that "task" is not an architectural mode — it's just which token types show up in the
stream. Transcription is audio-in, text-out. TTS is text-in, speech-tokens-out. A spoken reply is a
single sequence that switches from text to speech tokens partway through. Flip between them:

<UnifiedDecoder />

At the far end, the discrete tokens hit dedicated detokenizers: a **speech decoder** (XCodec2, with
a causal variant for streaming) reconstructs the waveform for TTS and speech-to-speech, and an
**audio decoder** (XCodec1 with an enhancement VAE) handles general text-to-audio. Because the
whole thing is "just an LLM emitting tokens," it stays compatible with standard LLM training and
inference infrastructure — no bespoke serving path for the audio head.

## The text tax, measured

Here is the claim that matters, and it's a claim you can only make with a table. Scored on
**text-only** benchmarks against other recent audio LLMs, Audex doesn't just avoid regressing — it
sits at or near the top of nearly every one:

<TextTax />

Read the paper's own table and the pattern is stark. Audex 30B-A3B posts **AIME 2025 91.2**,
**MMLU-Pro 78.9**, **GPQA-Diamond 74.9**, **ArenaHard v2 81.6**, **IFBench 77.8**,
**LiveCodeBench v6 85.3**, and a **1M-token context** with **99.4 / 83.4** on 256K/1M needle-in-a-
haystack — while audio models like Voxtral and MiMo-Audio show the tax plainly and even a strong
omni peer trails on reasoning and long context.

<Figure
  src="/articles/nemotron-audex/fig2.png"
  alt="Table of text-benchmark results comparing Step-Audio R1.1 33B, Voxtral Small-24B, MiMo-Audio 7B, Qwen3-Omni 30B-A3B Thinking, Qwen3.5-Omni Flash 35B-A3B, and Audex 30B-A3B and 2B across reasoning, knowledge, alignment, long-context, and agentic benchmarks."
  caption="Audex retains frontier text intelligence — reasoning, knowledge, alignment, 1M-token long context, and agentic tool use — where audio-tuned peers regress (paper, Table 5)."
/>

That "marginal or no regression" line in the abstract is the whole thesis. The audio ability is
additive, not a trade.

## What it does with the audio half

Keeping the text brain would be a hollow win if the audio were weak. It isn't. On the **OpenASR**
leaderboard (English), Audex 30B-A3B averages **6.82 WER** across eight test sets — 1.34 on
LibriSpeech clean, a table-best **1.76 on SPGI** — competitive with Whisper-large-v3 and the omni
models while being a single unified system:

<Figure
  src="/articles/nemotron-audex/fig3.png"
  alt="WER results on the OpenASR leaderboard across LibriSpeech clean/other, AMI, Earnings22, GigaSpeech, SPGI, TED-LIUM, and VoxPopuli, comparing Whisper, Canary, Qwen-Omni variants, and Audex 30B-A3B and 2B."
  caption="ASR word-error-rate on the OpenASR leaderboard; Audex is competitive with dedicated ASR models while also doing translation, understanding, TTS, and generation (paper, Table 8)."
/>

Around that sit speech translation, audio question-answering, text-to-speech and text-to-audio
generation, and — the one that closes the loop — **speech-to-speech**: spoken input to spoken
output in one model, no cascaded ASR→LLM→TTS pipeline with its latency and error stacking.

## How you buy no-regression

The recipe is where the tax actually gets dodged. Audex is trained on **157.4B audio tokens and
320.5B text tokens** — note the text is still the majority — through multi-stage supervised
fine-tuning, then a **text-only** Cascade RL pass plus multi-domain on-policy distillation.

<Figure
  src="/articles/nemotron-audex/fig4.png"
  alt="Audex training pipeline: two SFT curricula (multi-stage adding one capability at a time, versus a consolidated single-stage), followed by Cascade-2-style RL with MOPD to produce the final Audex model."
  caption="Two SFT curricula — capability-at-a-time vs consolidated single-stage — followed by text-domain RL and on-policy distillation, the step that keeps the text intelligence intact (paper, Figure 3)."
/>

Two details do the work. First, the SFT is studied as **two curricula** — a multi-stage one that
adds capabilities one at a time (text SFT → audio warmup → audio-gen → audio-gen + understanding)
and a consolidated single-stage one that mixes everything at once. Second, and more important, the
reinforcement-learning stage that follows is applied in the **text domain**, the same Cascade-2 RL
the backbone already knew. Audio is learned as *additional* token vocabulary on a preserved base,
and the final polish happens where the text intelligence lives — so it's reinforced, not eroded.

## The take

Audex's bet is that you don't need a clever fusion architecture to add audio to an LLM — you need to
refuse to treat audio as special. Encode it into the text embedding space on the way in, emit it as
extra vocabulary on the way out, keep the majority of your training tokens textual, and do your RL
where the reasoning is. The reward is a model that hears, speaks, and generates audio while still
scoring 91 on AIME and holding a million-token context. The "unified" in unified audio-text LLM
turns out to mean *boring on purpose* — and that's the compliment.

---

*Source: [Unified Audio Intelligence Without Regressing on Text Intelligence](https://arxiv.org/abs/2607.05196)
(Zhifeng Kong et al., NVIDIA) and the [model card](https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B).
Figures are the paper's; the interactive diagrams are mine.*