2026-07-20 · 5 min · audio · llm · multimodal · speech · explainer
There's a tax you pay when you make a text LLM multimodal. Bolt on a speech encoder, fine-tune for audio tasks, and the model's text scores — reasoning, knowledge, instruction-following — tend to sag. The audio ability arrives; some of the intelligence leaves. NVIDIA's Audex (Nemotron-Labs-Audex-30B-A3B, arXiv 2607.05196) is a unified audio-text LLM whose whole point is that it doesn't pay that tax. It does ASR, speech translation, audio understanding, text-to-speech, text-to-audio, and direct speech-to-speech — and keeps the frontier text scores of the model it was built on.
The design is almost aggressively simple, which is the interesting part.
One decoder, one vocabulary
Most audio LLMs treat audio as a side channel: an encoder produces features, an adapter head or a separate module consumes or emits them. Audex refuses the split. It is a single Transformer decoder built on Nemotron-Cascade-2-30B-A3B — a hybrid Mamba–Transformer mixture-of-experts, 30B total parameters with ~3B active, a 1M-token context. Audio inputs are turned into continuous embeddings by an audio encoder plus MLP adapters and dropped straight into the text embedding space. Audio outputs are discrete tokens drawn from an extended vocabulary (reported at 205,312 entries) that the model predicts in exactly the same autoregressive stream as text.
The upshot is that "task" is not an architectural mode — it's just which token types show up in the stream. Transcription is audio-in, text-out. TTS is text-in, speech-tokens-out. A spoken reply is a single sequence that switches from text to speech tokens partway through. Flip between them:
Nothing here is a bolted-on adapter head. The backbone emits speech and audio tokens from the same softmax it uses for text, in one autoregressive stream — so a spoken-in / spoken-out turn is just a sequence that happens to switch token types partway through. That uniformity is what lets Audex keep its text brain while gaining ears and a voice.
At the far end, the discrete tokens hit dedicated detokenizers: a speech decoder (XCodec2, with a causal variant for streaming) reconstructs the waveform for TTS and speech-to-speech, and an audio decoder (XCodec1 with an enhancement VAE) handles general text-to-audio. Because the whole thing is "just an LLM emitting tokens," it stays compatible with standard LLM training and inference infrastructure — no bespoke serving path for the audio head.
The text tax, measured
Here is the claim that matters, and it's a claim you can only make with a table. Scored on text-only benchmarks against other recent audio LLMs, Audex doesn't just avoid regressing — it sits at or near the top of nearly every one:
On IFBench Audex ranks #1 of 6. Across the table it holds frontier text scores — AIME 91.2, MMLU-Pro 78.9, a 1M-token context — where models that grafted audio on top give much of it back. The audio ability is additive, not a trade.
Read the paper's own table and the pattern is stark. Audex 30B-A3B posts AIME 2025 91.2, MMLU-Pro 78.9, GPQA-Diamond 74.9, ArenaHard v2 81.6, IFBench 77.8, LiveCodeBench v6 85.3, and a 1M-token context with 99.4 / 83.4 on 256K/1M needle-in-a- haystack — while audio models like Voxtral and MiMo-Audio show the tax plainly and even a strong omni peer trails on reasoning and long context.
That "marginal or no regression" line in the abstract is the whole thesis. The audio ability is additive, not a trade.
What it does with the audio half
Keeping the text brain would be a hollow win if the audio were weak. It isn't. On the OpenASR leaderboard (English), Audex 30B-A3B averages 6.82 WER across eight test sets — 1.34 on LibriSpeech clean, a table-best 1.76 on SPGI — competitive with Whisper-large-v3 and the omni models while being a single unified system:
Around that sit speech translation, audio question-answering, text-to-speech and text-to-audio generation, and — the one that closes the loop — speech-to-speech: spoken input to spoken output in one model, no cascaded ASR→LLM→TTS pipeline with its latency and error stacking.
How you buy no-regression
The recipe is where the tax actually gets dodged. Audex is trained on 157.4B audio tokens and 320.5B text tokens — note the text is still the majority — through multi-stage supervised fine-tuning, then a text-only Cascade RL pass plus multi-domain on-policy distillation.
Two details do the work. First, the SFT is studied as two curricula — a multi-stage one that adds capabilities one at a time (text SFT → audio warmup → audio-gen → audio-gen + understanding) and a consolidated single-stage one that mixes everything at once. Second, and more important, the reinforcement-learning stage that follows is applied in the text domain, the same Cascade-2 RL the backbone already knew. Audio is learned as additional token vocabulary on a preserved base, and the final polish happens where the text intelligence lives — so it's reinforced, not eroded.
The take
Audex's bet is that you don't need a clever fusion architecture to add audio to an LLM — you need to refuse to treat audio as special. Encode it into the text embedding space on the way in, emit it as extra vocabulary on the way out, keep the majority of your training tokens textual, and do your RL where the reasoning is. The reward is a model that hears, speaks, and generates audio while still scoring 91 on AIME and holding a million-token context. The "unified" in unified audio-text LLM turns out to mean boring on purpose — and that's the compliment.
Source: Unified Audio Intelligence Without Regressing on Text Intelligence (Zhifeng Kong et al., NVIDIA) and the model card. Figures are the paper's; the interactive diagrams are mine.