PhoneLLM Alpha 1: the fine-tune is free, the cost line isn't

Pipecat's phonellm-alpha-1 model card reads like a paper — a leaderboard, a latency budget, a cost estimator, an ablation-style before/after comparison — and it invites the same treatment a paper gets here: pull the primary sources apart and see what's actually load-bearing. There's no arXiv id attached to this one. The primary sources are the Hugging Face repo itself: config.json, the safetensors headers of thirteen shards, the six chart images the card embeds, and a linked Google Sheet that turned out to still be live and publicly readable.

The headline claim is straightforward — a full-parameter fine-tune of NVIDIA's Nemotron 3 Nano 30B-A3B, tuned specifically for phone-agent tool use, scoring "on par with GPT 5.6 Terra, but 94% cheaper and with 1,300ms faster P95 time-to-first-token" on a new benchmark the same team built, PhoneBench v1. Company benchmark, company model, company blog post — the discipline here isn't "assume it's wrong," it's "check what's checkable." Most of it is, and most of it holds up better than the median vendor claim on this site. One sentence in the middle of the cost section does not.

The 3.5B active checks out, tensor for tensor

config.json names the architecture: NemotronHForCausalLM, n_routed_experts: 128, num_experts_per_tok: 6, n_shared_experts: 1. The hybrid_override_pattern string — 52 characters, one per layer — spells out which of three block types each layer uses: Mamba2 mixer, full self-attention, or an MoE feed-forward block. Neither Mamba nor attention layers here carry a separate MLP; capacity comes entirely from the 23 layers typed E. Pulling the header of all 13 model-*.safetensors shards via HTTP range requests (the first 8 bytes give a header length, the next N bytes are a JSON manifest of every tensor's shape and dtype — no need to download 62GB of actual weights) gives the real split:

• 23 Mamba2 layers — in_proj / conv1d / out_proj, always active
• 6 attention layers — q/k/v/o_proj, always active
• 23 MoE layers, each: a router (gate.weight), 1 shared expert (always active for every token), and 128 routed experts stored as 128 separate tensors — experts.0.down_proj through experts.127.down_proj, not one fused block

Summing every tensor's element count gives 31,577,940,288 total parameters — 31.578B, against the header's own declared metadata of 31,577,937,344 (the ~3,000-element gap is rounding in a couple of F32 buffers, not a discrepancy worth chasing). Of that, 29.375B (93.0%) sits in the 128×23 routed-expert bank. Active parameters — everything always-on, plus 6/128 of the routed-expert mass — comes to:

dense (Mamba + attention + embed/lm_head/norms):  1,736,234,432
shared experts (always active, all 23 layers):       458,981,376
router gates (always active, all 23 layers):            7,916,416
routed experts, 6-of-128 active fraction:           1,376,944,128
                                                    ---------------
active total:                                       3,580,076,352

3.580B active, 31.578B total — the card's "30B total, 3.5B active" isn't rounded generously, it's accurate to the third significant figure. That's worth stating plainly, because the more interesting finding in this model card isn't a number that's wrong — it's what a correct number implies once you compare it against a model this is a fine-tune of.

The whole gain is the fine-tune, not the architecture

PhoneLLM is a full-parameter fine-tune — no adapter files, no LoRA config, no quantization markers in the repo, just a standard 13-shard bfloat16 checkpoint with the identical config.json as the base model. That single fact makes a specific comparison possible: the base model, NVIDIA-Nemotron-3-Nano-30B-A3B, appears in PhoneBench's own leaderboard as a scored row, under the same hardware, same serving stack, same everything except the weights.

Sort that table by latency or cost and PhoneLLM Alpha 1 and Nemotron 3 Nano 30B base land on the identical row: 331ms P50 TTFAT, 197ms floor, ~600ms P95, $0.0025/min — not close, identical to four decimal places. That has to be true if the fine-tune changed weights and nothing else (same tensor shapes, same MoE routing cost, same everything the GPU has to compute), and it's a clean, free sanity check that the benchmark's own latency and cost measurements are internally consistent rather than noise. What moved is the score: 28.6% for the untuned base, 72.3% for PhoneLLM — a 43.7-point jump, on a benchmark scoring tool-call accuracy and phone-agent speaking style, for the price of a training run that changed nothing about inference cost or speed. That's the actual headline result buried under the GPT-5.6 Terra comparison: full-parameter SFT bought PhoneLLM parity with a much larger frontier model's phone-agent behavior, without moving a single number on the latency/cost side of the ledger.

The before/after, in the model's own words

The card's qualitative example is a real transcript excerpt, not a synthesized illustration, and PhoneBench's judging methodology explains exactly what it's scoring when it looks at a turn like this one.

expand

Both examples score a specific, checkable behavior rather than a vague notion of "quality" — whether the greeting names the business, whether a response that claims to look something up actually calls a tool to do it. That's the exact axis the model card's own before/after example is built around:

expand

The failure mode is precisely the one the card's prose names: "LLMs will often say 'Yes, I've booked that table for you' without actually doing it." It's a real gap, not a strawman — the base model's sentence is a completely reasonable thing to say if the tool call had actually fired, and a customer-facing disaster if it didn't. Whether the judge panel that scores this is well-calibrated against the human labels the card describes is not something a model card alone can settle. What the model card's own transcript does settle is that this specific failure is real and reproducible on the identical two-turn conversation, and that the fine-tune fixes it on that conversation.

Where the 1,293 milliseconds actually go

expand

That table is worth adding up, because it's an easy place for a stray typo to hide and it doesn't have one: 40 + 21 + 10 + 2 + 40 + 1 + 300 + 650 + 20 + 120 + 21 + 2 + 10 + 40 + 1 + 15 = 1,293, matching the card's printed total exactly. That 650ms figure is a reference architecture's LLM time-to-first-token target (from a linked, separate voice-agent latency breakdown), not a PhoneLLM-specific measurement — and it's 50.3% of the whole voice-to-voice budget, the single largest of the fifteen stages by a wide margin. Every other stage is fixed by the audio pipeline (capture, codec, network, jitter buffering, playback) or the STT/TTS models sitting next to the LLM, not by which LLM is in the loop. That makes the LLM stage the one lever this card's own numbers say is worth pulling, and PhoneLLM's own leaderboard entry says it pulls harder than the reference target assumed:

Set the picker to PhoneLLM's own P50 and the pipeline totals 974ms, not 1,293 — because PhoneLLM's actual measured PhoneBench TTFAT (331ms, from the leaderboard) is well under the 650ms target the card's worked example assumed for that slot. Holding the same fixed 643ms and swapping in Claude Sonnet 5's P95 instead pushes the pipeline to 2,809ms, 87% over the card's stated 1,500ms voice-to-voice target — not because the audio pipeline changed, but because one number in a 15-number sum swung by over two seconds. The "1,300ms faster P95 time-to-first-token" claim against GPT-5.6 Terra also holds up against the leaderboard's own numbers: Terra's P95 is 1,957ms, PhoneLLM's is given as "~600ms," and 1,957 − 600 = 1,357 — close enough to "1,300ms" that the approximation in the card's own "~600" is doing the rounding, not an inflated claim.

94% cheaper, and the arithmetic that gets there

The cost side of the story starts from a real, confirmed number: Modal's published B200 rate is $6.25/hour, and the card's $6.2496/hour figure matches it. From there the card walks through a specific worked example — region pinning, target utilization, concurrency — to arrive at a cost-per-minute figure for running PhoneLLM on a dedicated Modal endpoint.

Every step up to the second-to-last is exact: $6.2496 × 1.5 (region multiplier) = $9.3744/hour; ÷ 0.70 (target utilization) = $13.392/hour, or $0.2232/minute — both numbers the card states and both correct. The last step — "Dividing $0.2232/minute by 88 concurrent agents equals a per-minute agent cost of $0.00025" — isn't: 0.2232 ÷ 88 = 0.002536, roughly ten times the printed figure. This isn't just a hand-recomputation disagreeing with prose. The card links its own cost-estimator spreadsheet, and that spreadsheet's own row for "PhoneLLM 30B Alpha 1" lists $0.0025/min — matching the correct division, not the $0.00025 sentence a few paragraphs above it. The leaderboard chart earlier in the same document lists the identical $0.0025 figure a third time. The error is contained to one sentence; everything that generates the correct number, and everything that would let a reader catch the mistake, is sitting on the same page.

The "94% cheaper than GPT-5.6 Terra" framing is close but not exact once the corrected $0.0025 figure is used: Terra's leaderboard cost is $0.0347/min, and (0.0347 − 0.0025) / 0.0347 = 92.8%, not 94% — a real but modest overstatement, about 1.2 percentage points, most plausibly from a slightly different snapshot of the same live, publicly-editable spreadsheet (its slider assumptions for conversation length and zero-data-retention pricing only affect the token-priced API rows, Terra among them, and are explicitly adjustable by anyone with the link).

What ships, and what the defaults actually do

The card is explicit about how to run this model: "set temperature to 0 and disable thinking. These two settings align with how the model was trained." Neither is what a naive load defaults to. generation_config.json ships "do_sample": true with no temperature key at all — which means an unmodified model.generate() call samples at the library default of 1.0, not the recommended greedy temperature=0. The chat template's own Jinja carries {%- set enable_thinking = enable_thinking if enable_thinking is defined else True %} — thinking is on by default unless a caller explicitly passes chat_template_kwargs: {"enable_thinking": false}, which is exactly the override the README tells you to pass and exactly the setting that isn't the file's own default. The repo does ship the plumbing for the recommended path correctly — a custom nano_v3_reasoning_parser.py for vLLM specifically handles the case where thinking is disabled and the model's answer would otherwise land in the wrong field — so the "right" configuration is fully supported. It just isn't what you get by not reading the README.

The license is a genuine two-layer arrangement, not just a BSD stamp: PhoneLLM's own terms are BSD 2-Clause, but it's a derivative of NVIDIA's Nemotron 3 Nano checkpoint, which ships under the NVIDIA Nemotron Open Model License. That upstream license's Section 3 (Redistribution) survives the fine-tune and binds anyone who redistributes PhoneLLM or a derivative of it: include a copy of the NVIDIA license, retain NVIDIA's copyright/attribution notices, and — if the work carries a NOTICE file — propagate an attribution statement crediting NVIDIA. BSD 2-Clause governs Pipecat's own modifications and the model as a whole, as the NVIDIA license's Section 3 permits, but it doesn't erase the upstream obligations; both licenses apply, layered rather than replaced.

The ledger

What holds up: the 3.580B-active / 31.578B-total math, to three significant figures, verified from the raw tensor shapes rather than taken from the README. The 1,293ms latency-budget sum, added by hand, exact. The base-model comparison, which is really the strongest evidence in the whole card — a fine-tune that changes score by 43.7 points while provably changing nothing about cost or latency, because the two models are computing on the same graph. The "1,300ms faster P95" claim, which survives being checked against the leaderboard's own numbers. What doesn't: one arithmetic step in the cost walkthrough, off by roughly 10x and contradicted twice on the same page — a real error, but a narrow and easily-fixed one, not a sign the underlying cost numbers are made up. And a "94% cheaper" headline that rounds up from a real, still-compelling 92.8%.

None of this changes the shape of the actual result: a 30B-parameter, 3.5B-active MoE, full-parameter fine-tuned for phone-agent tool use, scoring within a point of a much larger frontier model on a benchmark built specifically to catch the failure mode voice agents actually have — while costing about a fourteenth as much per minute, at a P50 latency low enough that the LLM stops being the tightest part of the voice-to-voice budget. That claim was checkable, and it checks out. The one line that didn't just happened to be the one nobody re-derived before publishing.

More on the pieces underneath: the Mamba-Transformer-MoE hybrid architecture PhoneLLM inherits shows up again in TwoTower, also a 30B Nemotron hybrid; the sparse-routing mechanics generally are in Mixture of Experts, from scratch; the say/do consistency problem gets a very different fix in VoiceMem's retrieval-side memory architecture; and the serving stacks the card recommends are covered in vLLM and SGLang. The overlapping-stage streaming design in the voice loop itself is close kin to the turn-detection tricks in speech-to-speech.