Ming-3.0 speech models: one 0.5B speech engine inside five checkpoints, run on a CPU

On 19 October Ant Ling posted five open speech models from inclusionAI in one go: "Ming-3.0-tts: 0.5B voice design & paralinguistic cues; Tiny 7.9B A1.3B long-form speech. Ming-3.0-audio: Flash 126B A6.5B understanding & tools; Tiny 9.3B A2.7B efficient dialogue. Ming-3.0-asr: Tiny 8.6B A2B dialects & context." Satyajit likes real samples, so the plan was: read all five, run the one that fits on a CPU, and listen.

The smallest one turns out to be the key to the rest. Ming-3.0-tts-0.5B is a complete speech engine. The two audio models carry the same weights, byte for byte, as their voice. So this piece explains that engine first, then shows what was built around it.

The release, file by file

Every repository is on Hugging Face under inclusionAI/, and every card says MIT. I listed them through the Hub API, read each config.json, and pulled each safetensors header with an HTTP range request, which gives every tensor's name and shape without downloading the weights. The audio models also ship talker/data/lid.176.ftz, fastText's language identifier, whose bundled note says it is CC BY-SA 3.0. The ASR repo bundles Apache-2.0 files and Whisper's MIT notice. So "MIT" is right for the weights and the code Ant wrote, and the repos as shipped are mixed.

There is no technical report. Each repo has a PDF named …-TDS2026.pdf, which is the EU AI Act's public summary of training data, not a paper. It ticks "more than 1 million hours" of audio and "more than 10 trillion tokens" of text, and for a text-to-speech model it says the corpus "is designed for a text-output model": template text nobody edited. The Ming-3.0-tts README's roadmap still has "Technical report" unticked. Everything below comes from configs, code and headers.

How Ming-3.0-tts writes sound

Start with the problem. A language model predicts one token from a vocabulary. Speech at 44.1 kHz is 44,100 numbers a second. The usual answer is a codec that turns audio into discrete codes, and a language model that predicts codes; Fish Audio S2 and Qwen3-TTS work that way. Ming does not quantize. Its tokens are continuous.

The audio tokenizer is a VAE. audio_tokenizer/vae_modules.py cuts the waveform into 882-sample frames, 20 ms each. A 24-layer transformer with the Qwen2-0.5B shape reads them, a 4-layer aggregator squeezes every 4 frames into one, and a linear layer emits a mean and a variance for a 64-number latent. So one latent covers 80 ms, 12.5 a second; the README's figure calls it "Tokenizer-12.5Hz" (measured, from hop_size 882 and patch_size 4). The decoder runs the other way: upsample 4x, a second 24-layer transformer, and an inverse-STFT head with a 3,528-point FFT and an 882-sample hop.

The language model writes one latent per step. The backbone is Qwen2.5-0.5B: 24 layers, 896 wide (measured). Its prompt is ordinary chat text: a system line, the words to say, then an assistant turn holding the controls, then a special <audio> token. From there modeling_bailingmm.py loops:

1. Run the backbone one position forward and take the last hidden state.
2. Hand it to the DiT patch head, an 8-block, 1,024-wide transformer, which draws one 64-number latent by flow matching: start from noise, take 10 Euler steps along the learned velocity. Each step runs twice, with and without the hidden state, and mixes them at guidance 2.0. At the default temperature 0 there is no noise after the first draw, so a fixed seed gives a fixed take.
3. Ask a two-way stop head whether speech is over; it must say yes with more than 0.5 and only after step 3.
4. Pass the latent through the aggregator, another 8-block transformer, into an 896-number embedding, and feed that back in as the next input.

When the loop stops, the decoder turns the whole run of latents into a waveform in one pass. The code comments say why: chunked streaming decode "produces slightly different audio at the chunk boundaries." The reference runtime has a static 512-position cache, so a fragment can run for at most 512 minus the prompt length steps. That is under 41 s of audio, before the prompt takes its share (reasoned). Longer text is cut into fragments at punctuation.

expand

Here is what that costs. Every step runs the 0.5B backbone once, the 103M-parameter DiT head 20 times (10 steps, twice each), and the 102M aggregator once. On my run that was about 0.43 s a step at a short context, against 0.08 s of audio per step (measured: 72 steps, 31.4 s of the 32.5 s call).

What the 0.5B really is. The header of model.safetensors holds 1.4795B parameters (measured). The backbone with its tied embedding is 494.0M; that is the "0.5B". The VAE encoder is 419.3M, the decoder 361.1M, the DiT head 103.1M and the aggregator 101.7M. A reply under the post said this: "the smallest tts is dense on a qwen2.5 backbone with a dit head, audio vae, and cam++ speaker encoder packed into one folder." That holds; the CAM++ speaker encoder is a separate 28 MB campplus.onnx. The file is 2.96 GB in bf16. Without a reference clip, a generation uses about 1.06B of the 1.48B: everything except the encoder (reasoned).

Voice design and tags are only text

The interesting controls are the cheapest to explain. tts_tasks.py turns every structured control into a JSON caption that goes on the assistant's side of the prompt, after the text and before <audio>. Its keys are Chinese and fixed: 序号 (index), 说话人 (speaker), 方言 (dialect), 风格 (style), 语速 (speed), 基频 (pitch), 音量 (volume), 情感 (emotion), a BGM block, and IP, the built-in voice name. Voice design writes your description into 风格. A built-in voice writes its name into IP. Freeform puts your instruction there as plain text instead. The model learned all of these as captions during fine-tuning, and the code warns that "the strings and JSON key order in this module are part of the model input protocol."

The 14 paralinguistic tags ([Laughter], [Sigh], [Cough], [Breathing] and ten hesitation and question sounds) are not special tokens either. I ran the shipped tokenizer: [Laughter] is four ordinary BPE pieces, [, La, ughter, ], and [Dissatisfaction-hnn] is six (measured). The only code that knows about tags is the text frontend, which routes around them so normalization does not turn [Laughter] into "，Laughter，". Another reply guessed that at this size "the paralinguistics are living in the codec rather than the language model." The code says the opposite: the codec is a plain VAE with no idea what a tag is; the tag is text the language model reads, and the laugh comes out as latents like any word does. This is the same design Fish Audio S2 uses, and Fish Audio Drama 3 takes it further with stage directions in prose.

What I ran, and what I heard

Inference needs the GitHub repository; the HF repo holds only the checkpoint. I cloned it at commit 028c1ea, made a Python 3.12 venv with the CPU wheels of PyTorch 2.6.0 and its pinned transformers==4.46.3, and called its MingTTS.synthesize from a small driver. The machine is a shared cloud VM with 4 cores, 15 GB of RAM and no GPU, and five other authors were running jobs on it. I used torch.set_num_threads(2) under nice -n 19; the load average sat between 8 and 20. Read every time below as a ceiling for this CPU.

Two things needed working around. First, disk: the machine had about 4 GB free, shared. The VAE encoder only runs when you clone a voice from a reference clip, and its tensors sit in one contiguous 838.5 MB byte range of the file (measured). I downloaded the file with that range left as a sparse hole that reads as zeros, so cloning was off the table and every take below uses a built-in voice, a description or an instruction. Second, the freeform path crashed on the CPU with mat1 and mat2 must have the same dtype, but got Float and BFloat16. The code wraps generation in torch.autocast(device_type='cuda'), which does nothing on a CPU, so the fp32 zero speaker embedding meets bf16 weights. A one-line hook that casts that input fixed it. A separate CPU write-up linked under the post, ming-tts-on-cpu, hit the same error on every call at an older commit; at 028c1ea only the zero-embedding path still trips it. pynini was not installed, so text normalization was off and I kept digits out of the texts.

The same takes as a table. "Made in" is wall time for the call after load, RTF is that over the audio length, RAM is peak resident memory during the call, and the transcripts are openai/whisper-base reading the WAV back (all measured).

The English is word-perfect to Whisper on all four English takes. The Chinese errors are 声称 for 生成, 刻 for 颗 and 语 and 听 for 雨 and 停: every one a near-homophone, and Whisper-base is a small recognizer in Chinese, so I would not pin them on the TTS without a better ear. I could not run Ming's own recognizer; its section says why.

The runs are deterministic at 2 threads. I repeated take 1 and take 4, and both came back bit-identical (measured), while the wall time of take 1 went from 32.53 s to 77.05 s because the machine got busier. The CPU write-up above saw different audio on every call at 8 threads; fewer threads, fewer reductions in a different order.

Take 4 is the flaw. I gave voice design the cookbook's own Chinese description of an ASMR whisper: very soft, breathy, close to the microphone. What came out is not a whisper; at −22.0 dBFS it is louder than both plain built-in voices (−23.9 and −25.9 dBFS) and within 0.2 dB of the loudest take (measured). It also took 363 s, and 789 s on the repeat. With silence trimming off and a step counter on the flow head, the reason is plain: the stop head never fired. The loop ran 304 steps, 24.32 s of audio, until the 512-position cache was full (the long description took the other 208 positions). After 6 s everything is near-digital silence, peak 0.00006, and the trimmer threw away 18.6 s of it (measured). The English voice-design take, by contrast, stopped by itself after 79 steps. One seed is one seed; the card does say open-ended voice design "does not yet match the strongest evaluated system."

Did the tags do anything? I generated takes 5 and 6 again with the tags removed, same voice and seed. With [Laughter] the take is 5.40 s against 3.50 s without, and it is the quietest of the seven at −30.8 dBFS, its level broken up through the middle where the plain take is solid speech. With [Sigh] and [Cough] the take is 4.10 s against 3.50 s, with a short burst and a gap before the first word and a second gap where the cough sits (measured, from 50 ms loudness envelopes). That is evidence the tags change the audio where they are. Whether it sounds like a laugh is for your ears; Whisper writes no laugh.

Ming-3.0-tts-tiny: the same loop on a sparse backbone

The tiny TTS keeps the loop and swaps the backbone for Ling V3, the hybrid MoE behind Ling-3.0-flash. Its config: 24 layers, 1,536 wide, every fourth layer gated multi-head latent attention and the other 18 Kimi Delta Attention, a linear-attention recurrence. Each MoE layer has 128 experts plus one shared, and routes each token to 8. The DiT head doubles to 16 blocks; the VAE is the same shape. The repo holds 8.98B parameters (measured). The card's 7.9B is the backbone alone: embedding, head, attention and experts sum to 7.893B. Its "about 1.3B" active sits between 1.14B without the input embedding and 1.38B with it (reasoned).

I did not run it: 18.0 GB of weights, and its reference environment is CUDA 11.8 with Triton kernels for KDA. A reply under the post points at mlx-Ming-3.0-tts, which rewrites the KDA recurrence for Apple's MLX and publishes 4-bit and 8-bit conversions; I read it but had no Mac to run it on. "Long-form speech" in the post maps to the podcast task, a script with speaker_N: lines, and the card's own limitation: podcast UTMOS 2.12, below ZipVoice-Dia's 2.24 (reported).

The audio models wrap the TTS

Ming-3.0-audio-flash and -tiny read audio and write text, and optionally speak the answer. The input side is a Whisper-large-v3-shaped encoder (32 layers, 1,280 wide, 128 mel bins, 638.9M parameters, measured), a small MLP projector, and a Ling V3 backbone. Flash's backbone is 42 layers, 2,560 wide, in 7 groups of 5 KDA layers and 1 gated MLA layer, with 512 experts and 8 routed per token; audio-tiny's is the TTS-tiny's shape. Both also ship a multi-token-prediction layer, 3.07B parameters in flash, that the headline leaves out.

The output side is a folder called talker/. Its config is the TTS's shape, and its weights are the TTS's weights. I hashed eight tensors from both audio repos against Ming-3.0-tts-0.5B: the stop and speaker heads, a backbone MLP, a DiT block, an aggregator block, a VAE decoder layer and an attention bias. All eight match byte for byte (measured). So when flash speaks, it is the 0.5B model above, reading flash's text.

expand

The headline sizes need a key, which is what the ledger above is for. Flash's 126B reproduces as backbone plus audio encoder plus Talker without the MTP layer: 125.8B (reasoned from measured parts). Audio-tiny's 9.3B and 2.7B reproduce by the same rule, at 9.25B and 2.74B. Apply that rule to flash's active count and you get 6.9B, not 6.5B. The only cut I found that gives 6.50B drops the input embedding, which audio-tiny's count keeps. Neither is wrong; they are two bookkeepings on one launch slide. Serving flash, per its card, takes four 80 GB H800s for the 128K profile.

Ming-3.0-asr-tiny: context in the prompt

The recognizer is the audio-tiny without a voice: the same Whisper-shaped encoder, a projector, and the 24-layer Ling V3 MoE. It totals 8.55B without its MTP layer and 2.04B active (measured), against "8.6B A2B" on the card. Context ASR is a prompt, not a module: you append "This is an audio about …" and "This audio may contains the following words or phrases: …" to the instruction, and the model reads them as text.

expand

The "4.41% average error across four ASR categories" holds as arithmetic: 1.67, 8.42, 3.55 and 3.99 average to 4.41, and the other three models' overall bars check out the same way (reasoned). Read the categories, though. Domain and dialect are in-house test sets. The mean mixes CER and WER. Fun-ASR 1.5 is better on dialects (8.00 against 8.42) and FireRedASR2 on standard Chinese (2.94 against 3.55); Ming wins the average by winning domain and English.

The card's architecture figure is another model's. It labels the embedding 2,560 wide, the experts "E512A8" and the size "7.9B total parameters, with only 1.3B activated". The ASR config says 1,536 wide and 128 experts, and 7.9B and 1.3B are the TTS-tiny's numbers (measured against config.json).

expand

I did not run it. It is 17.7 GB in bf16, and its example asks for transformers 5.12.1, PyTorch 2.11 and a CUDA GPU.

The leaderboard claims

The post's TTS numbers are from Artificial Analysis: "#1 open-weight model in AA Controlled Voice (1069 Elo)", "#3 on Provider Voice (1188 Elo)", and "Ties #1 among all models on Pronunciation Robustness (93.2%)". The README's own screenshot shows all three, and shows their margins. On Controlled Voice the rank range is 1-2 against Fish Audio Drama 3 Preview at 1059. Pronunciation is 93.2% for both of them. Neither the post nor the README says which checkpoint, 0.5B or tiny, was entered as "Ming-3.0-tts" (reported; unverified).

expand

The card's own tables are more mixed than the slide. On Seed-TTS the 0.5B has 0.800% WER in Chinese and 1.584% in English; CosyVoice3-1.5B has 0.710% and Qwen3-TTS-12Hz-1.7B 1.240% (reported). Where it leads is control: NVBench single-tag Chinese CER 2.70, dialect CER 2.27% on WSC-Eval-TTS-easy, pitch control 86.00% (reported).

Where this leaves it

For a CPU-only service, the 0.5B is usable as a batch voice, not a live one: 5.6 times slower than real time on 2 busy threads, under 2.7 GiB of RAM, deterministic, and correct to Whisper in English. The speech.cpp build of Qwen3-TTS 0.6B ran 3.7 to 4.0 times slower than real time on this same VM class, and Paradee shows how far down a voice can be squeezed when it does not need controls. The voice-design path needs a guard: cap the steps yourself, because a description long enough to fill the cache can leave the stop head silent.

The design is the takeaway. One 0.5B loop, continuous latents at 12.5 a second and controls written as text, is the speaking part of all five releases. Everything bigger is a better ear in front of it.

The parameter counts on the cards below are Hugging Face's own tally of each repository's safetensors files, so they include whatever ships in the repo (an MTP layer, a Talker, the audio encoder) and will not always match the headline names or the ledger above, which separates those parts.