A grammar guarantees it parses. A menu guarantees it was on the menu.

2026-09-22 · 15 min · structured-generation · constrained-decoding · agents · inference · llm · explainer

XGrammar-2 is the constrained-decoding layer under SGLang, vLLM, TensorRT-LLM and MLC-LLM, and its headline feature is Structural Tag — a small JSON DSL that describes an output shape, compiles to a grammar, and masks the model's vocabulary at every decode step so the emitted tokens cannot leave that shape. Twenty-two format types, nineteen built-in model formats, and one response_format field that serving engines expose over the OpenAI API.

It is also the other half of an argument this site has been having with itself.

RLCD is not constrained decoding separated two things people conflate: a decoder-side constraint on what a model may emit, and a training-side property of how well its probabilities track reality. What a decision model cannot do then showed what the first choice costs — a model whose output is always an element of the caller's option set cannot write a string, and needs a generative model beside it to do so.

XGrammar is the third position. Do not replace the generator. Constrain it.

So: what does each one actually promise?

two answers to “stop the model emitting something my program cannot handle” · neither guarantee implies the other

A grammar guarantees membership in a language; a menu guarantees membership in a set. Neither implies the other: a language has infinitely many strings and the grammar does not care which one you get, while a set has no syntax and nothing stops its members being unparseable if you never checked. They coincide in exactly one place — when the language is finite and written out — and every practical schema is a mixture: enums where the values are known, free strings where they are not. The free strings are the residue neither mechanism validates.

The two guarantees, stated precisely

A grammar guarantees membership in a language. The masking is exact: a token whose emission would take the prefix outside the language is assigned zero probability, so there is no decode path to an invalid string. If your language is "a JSON object matching this schema", you will get a JSON object matching that schema, with probability one, forever. What you will not get is any promise about which member of that language you got — a schema with one free string property admits uncountably many documents and the grammar is indifferent between all of them.

A menu guarantees membership in a set. The output is an element of the option list the caller supplied on this request. It cannot be a value the caller did not offer, cannot be a hallucinated tool name, cannot be an enum member from last year's schema. What it will not give you is a value nobody enumerated, and the docs of the model that sells this hardest say so plainly: "not trained to generate text. While you can force it to by chaining choices, this will not work well and will be very slow."

Neither implies the other, and both directions of that matter.

A grammar does not imply a menu: to guarantee "the value is one of these five", you must write the five into the grammar. Then you have a menu — which is a real construct here, not a rhetorical one, and I will come back to it.

A menu does not imply a grammar: nothing about "choose one of these options" guarantees the chosen option is syntactically anything. It is well-formed only if every option you supplied was well-formed, which is a property of your code, not of the model.

The evidence for this is XGrammar's own benchmark

The cleanest demonstration that these come apart is in the post's first figure, and it is the figure that should be quoted from it.

Two bar charts side by side on BFCL-V3, with and without a structural tag, for five models: Llama-3.2-1B, Llama-3.2-3B, Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen2.5-72B. The left chart is schema accuracy: without the tag the values run from about 21 percent for the 1B up to 100 percent for Qwen2.5-72B; with the tag every bar is exactly 100 percent. The right chart is output accuracy: without the tag it runs from about 5 percent for the 1B to about 92 percent for Qwen2.5-72B; with the tag the 1B rises to about 46 percent, the 3B to about 88, Qwen3.6-27B to about 93, the 35B-A3B to about 90, and Qwen2.5-72B is essentially unchanged and a hair lower. — Read the two panels together. Left: schema accuracy goes to exactly 100% for every model — that is the guarantee, delivered, and there is nothing probabilistic about it. Right: output accuracy does not. The 1B goes from 5% to about 46%, which means most of the 79 points of unparseable output became parseable and wrong. (blog.mlc.ai, 'XGrammar-2: Fast and Customizable Structured Generation for Tool Calling and Agents', 4 May 2026, Figure 1, flattened onto white.)

Every output is in exactly one of three states — correct, parses but is wrong, or does not parse — and the two panels pin all three. Decomposed:

BFCL-V3, simple and parallel subsets · every tool call is correct, or parses and is wrong, or does not parse

The grammar does exactly what a grammar guarantees: the red segment goes to zero, every time, for every model. What it does not guarantee is where those calls land. 79 points of Llama-3.2-1B’s output stopped failing to parse, and about 41 of them became correct — the other 38 became tool calls that parse cleanly and are still wrong. At the other end, Qwen2.5-72B had no red segment to remove and the tag changed its score by a tenth of a point, downward. Values read off the published bars, so treat them as approximate.

The grammar deletes the third state completely. It splits the deleted mass between the other two, and the split is not in its control. For Llama-3.2-1B roughly half the recovered points became correct answers and half became tool calls that parse cleanly and call the wrong thing. For Qwen2.5-72B, which already conformed, the tag changed the score by a tenth of a point downward.

This is the right result and the post is honest about it: "XGrammar is best used to enforce format constraints, not to change the semantics of an LLM's response. It helps downstream programs avoid fatal failures from malformed outputs." Fatal failures, not wrong answers. A malformed tool call throws; a well-formed wrong one runs.

Where the two mechanisms are literally the same construct

The interesting part of the comparison is that it has a fixed point, and it is in XGrammar's type list rather than in anyone's argument.

{ "type": "or", "elements": [
  { "type": "const_string", "value": "refund" },
  { "type": "const_string", "value": "replace" },
  { "type": "const_string", "value": "escalate" }
]}

That is a menu, written as a grammar. OrFormat over ConstStringFormat, both first-class types in python/xgrammar/structural_tag.py. The equivalent through JSON Schema is an enum, and XGrammar's C++ converter has a name for exactly this case:

// cpp/json_schema_converter.cc
bool TryGetFiniteValues(const picojson::object& schema, std::vector<picojson::value>* values) {
  if (schema.count("const")) { values->push_back(schema.at("const")); return true; }
  if (schema.count("enum"))  { /* ... */ return true; }
  return false;
}
// and, above the one-of classifier:
struct OneOfArmProof { enum class Kind { kTypeSet, kFiniteValues }; /* ... */ };

kFiniteValues is the grammar compiler noticing that a fragment of your schema is a menu. When it fires, the grammar's guarantee and the decision model's guarantee are the same guarantee, and the token-mask arithmetic is the same arithmetic — restrict the vocabulary to the legal continuations, renormalise, sample.

Which is the useful way to think about a real schema: it is a mixture. Some fields are finite and both mechanisms bound them. Some are not, and neither does.

one search_products tool call · what a grammar bounds, and what a menu can supply

The two shaded rows are the whole disagreement. A grammar will happily emit "query": "" or a paragraph of nonsense — both are quoted runs of characters, so both parse. A decision model will not emit either, because neither was on a list. That is why the constructive answer is not one mechanism but a split: enumerate the fields you can, let the grammar bound the shape of the rest, and know which ones are residue, because those are the only ones where a wrong answer looks exactly like a right one.

The residue is where the two mechanisms fail in different ways and both fail. A grammar will emit "query": "" without complaint, because an empty quoted run is a string. A decision model will not emit it, because it was not on a list — which is not better, it is just a different failure. The 379-of-385 figure from the WindTunnel traces is the empirical shape of that residue in a real agent: nearly every action a bounded model selected had at least one field it could not fill.

What Structural Tag actually adds over JSON-schema decoding

Constrained decoding against a JSON schema is old. The specific thing XGrammar-2 adds is that the schema and the wire syntax are separated.

A left-to-right flow diagram. A user talks to an agent application, which exchanges messages with a tool-calling box above it. The agent application sends a Structural Tag — shown as a JSON document declaring a tag with begin '<answer>', content of type json_schema with status and message string properties, and end '</answer>' — to a serving engine on the right, labelled with SGLang, vLLM and TensorRT-LLM marks. Inside the serving engine, XGrammar 2 constrains the model output to follow the structure. The result flows back as a structured output, shown as the literal text '<answer>' followed by a JSON object with status success and a message, then '</answer>', and returns to the agent application. — The separation that makes this more than JSON-schema decoding: the caller declares a shape, the engine compiles it, and the model's vocabulary is masked against it. The schema in the middle describes the arguments; the begin and end markers describe the wire format, and they are independent. (blog.mlc.ai, 4 May 2026, Figure 4, flattened onto white.)

DeepSeek V4 does not emit tool calls as JSON. It emits this:

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Your tool's schema is still {"city": {"type": "string"}}. The surface syntax is XML-flavoured with model-specific control tokens. Under a JSON-only constrained decoder those are two incompatible facts; under Structural Tag they are a json_schema node with a style:

`style`	what the arguments look like on the wire
`json`	standard JSON
`qwen_xml`	`<parameter=key>value</parameter>`
`minimax_xml`	`<parameter name="key">value</parameter>`
`deepseek_xml`	`<{dsml}parameter name="key" string="true\|false">value</{dsml}parameter>`
`deepseek_v4_1_xml`	as above, with a space after the control token
`glm_xml`	`<arg_key>key</arg_key><arg_value>value</arg_value>`
`cohere_xml`	`<cofl:value name="key" type="raw\|json\|dict\|list">value</cofl:value>`
`kimi_k3_xml`	`<\|open\|>argument key="key" type="type"<\|sep\|>value<\|close\|>argument<\|sep\|>`
`minimax_m3_xml`	recursive namespace XML

Nine wire formats, one schema language. That is a lot of accidental complexity to absorb, and absorbing it in one library rather than in nine serving-engine tool parsers is the actual contribution. The post's framing agrees: "Supporting all of these requires significant effort from serving engines and downstream applications, and may still fail to match the official specification."

The second thing it adds is composition. TriggeredTags lets the model write free text until it emits a trigger string, at which point the output must follow a structured tag — which is how you express "reason freely, then call a tool strictly" without constraining the reasoning. That is a shape a JSON schema cannot describe at all, and it is the shape every reasoning model now has.

The costs, which are real but small

Two charts. The left is tool-calling compilation time on a logarithmic vertical axis in milliseconds, against total tool count of 10, 100 and 500. XGrammar rises from 783.60 milliseconds to 6,697.0 to 60,308.0; XGrammar-2 rises from 45.56 to 173.33 to 731.73. The right chart is average end-to-end latency in seconds in SGLang for Llama-3.2-1B, Llama-3.2-3B and Llama-3.1-8B, with and without a structural tag. The 1B goes from about 0.265 to about 0.260 seconds, the 3B from about 0.157 up to about 0.166, and the 8B from about 0.258 to about 0.256. — Left: the 80x claim, which is 82.4x at 500 tools (60,308 ms to 731.73 ms) and 17.2x at 10. Right: per-request latency overhead, which is signed both ways — the 3B pays 5.7% and the other two save 1 to 2%. Note the qualifier in the post's own text: 'in a single-batch setting with warm-up and comparable output lengths'. (blog.mlc.ai, 4 May 2026, Figure 2, flattened onto white.)

Compilation. Fifty tools of JSON schema used to be a 6.7-second stall. The cross-grammar cache (an automaton-based hierarchical hash that finds shared sub-structures — "nearly 50% of structures end up reused" at 50 tools) and repetition-state compression ({"maxItems": 1000000} from $O(n)$ to $O(1)$ , 534 ms to 5.37 ms) take 500 tools from 60.3 seconds to 0.73. That is the 80×.
Per token. Near zero, and signed both ways in the published bars. The mask comes from a precomputed token-mask cache, so the common case is a lookup.
Against speculative decoding. This is the one that would bite, and it is the one the post spends most engineering on. A draft tree has to be masked node-by-node on the CPU while the GPU verifies it; done serially that is a stall per round. traverse_draft_tree walks the whole tree once and emits masks for every node, so the CPU work overlaps the GPU verify. Given how much of modern serving is now speculative, a constrained decoder that serialises against the draft verify is not deployable, and this is the fix.

What this does not settle

The comparison has a limit worth stating, because the two mechanisms are not substitutes for each other on any axis except "stop the invalid output".

Constrained decoding does not calibrate. Masking renormalises the softmax over the legal subset, which is exact arithmetic and says nothing about whether the resulting probability means anything. The subset softmax is exact and it is not calibrated, and that is precisely the distinction RLCD's name collision obscures: a decoder-side constraint and a training-side calibration property are orthogonal, and you can have either, both, or neither.

A decision model does not parse. It returns an index and a probability. If you want JSON out of it you write the JSON yourself, from the option you selected, which is trivially correct and is the reason that architecture never has a parse bug.

Nothing here judges the semantics. Both mechanisms narrow the space and neither ranks what is left. The 46.5% output accuracy on a constrained Llama-3.2-1B is a statement about the model, and no grammar is going to move it.

The honest composition, then, is the one the WindTunnel repository arrived at by necessity and the one XGrammar's type list supports directly: enumerate every field you can, constrain the shape of the rest, and keep a list of which fields are residue — because those are the ones where a confidently wrong answer is indistinguishable, to your program, from a right one.

What I would take away

"100% schema accuracy" and "the model got it right" are different claims and the same figure carries both. Ask which one a structured-output vendor is quoting.
A grammar's guarantee is unconditional and narrow. Zero probability on invalid continuations is not a 99%; it is a proof. That is worth a great deal, and it is worth exactly nothing about correctness.
Look for the enum. Every place a schema enumerates its values is a place the two mechanisms coincide and both guarantees apply. Every free string is a place neither does. That audit takes ten minutes and tells you where your agent will fail quietly.
Check the speculative-decoding path before adopting a constrained decoder. If masking does not overlap the draft verify, constrained decoding costs you the speculation, which on current serving stacks is a larger number than anything the grammar saves.

Related reading: why RLCD is not constrained decoding; what a decision model cannot do, whose WindTunnel split is the residue measured in the wild; parallel constrained decoding from first principles, which builds the mask in forty lines; and SGLang's jump-forward decoding, which is the same compiled-FSM machinery used for a different purpose and is currently dead code.

What would change my mind

6 claims above, and what would falsify each

Constraining Llama-3.2-1B took schema accuracy from ~21% to 100% and output accuracy from ~5% to ~46.5%, so roughly half the recovered mass became parseable-and-wrong.
Read off the two bar charts in the post's Figure 1, to the nearest half point, which is the weakest link in this article. The decomposition assumes output accuracy counts a subset of schema-conforming calls — that a call scored correct must also conform. If BFCL-V3's output-accuracy metric can credit a call that fails schema validation (via a lenient parser, say), the three states are not disjoint and the arithmetic is wrong. The post's evaluation scripts are linked from it; running them would settle both the readings and the metric definition.
A grammar and a menu coincide exactly where the grammar enumerates a finite set, and XGrammar implements that case explicitly.
python/xgrammar/structural_tag.py defines OrFormat and ConstStringFormat as first-class types; cpp/json_schema_converter.cc has TryGetFiniteValues, which returns true for const and for a non-empty enum, and an OneOfArmProof whose kinds are kTypeSet and kFiniteValues. Read at commit 40ef651. If those are an optimisation pass rather than the semantics — that is, if a schema enum compiles to something that admits values outside the list on some path — the equivalence is weaker than I have claimed, though a grammar could still be written by hand to enforce it.
Structural Tag's real contribution over JSON-schema constrained decoding is separating the schema from the wire syntax.
JSONSchemaFormat.style takes nine values in structural_tag.py, and builtin_structural_tag.py registers nineteen model formats. That is evidence of the feature existing, not of it being the most valuable one. A reasonable person could rank the cross-grammar cache higher — 60.3 seconds to 0.73 at 500 tools is a deployability threshold, not a percentage — and I would not argue hard.
A grammar guarantees the output parses but cannot make it correct, and the post agrees.
The post's own sentence: "XGrammar is best used to enforce format constraints, not to change the semantics of an LLM's response." If someone demonstrates a grammar construction that reliably raises semantic accuracy beyond eliminating format failures — by pruning branches that lead to known-bad continuations, say, which is a thing a grammar could in principle encode — then the guarantee is broader than I have drawn it. The BFCL numbers are the evidence against; they are five models on two subsets.
Per-token overhead is near zero, and signed both ways.
The post's SGLang latency bars show Llama-3.2-3B slower with the tag (about 0.157 to 0.166 s) and the 1B and 8B marginally faster. The qualifier is the post's own: "in a single-batch setting with warm-up and comparable output lengths." Batch serving with cold grammars and long tool lists is a different regime and nobody has published it. If the mask lookup contends at batch 256 the way the compile step used to contend at 500 tools, "near-zero overhead" is a single-stream claim only.
The residue — fields neither mechanism bounds — is most of a real agent's tool arguments.
This site's WindTunnel read: 385 of 413 selected actions needed a second model, and 379 of those carried at least one string property with no enum and no const. One benchmark, one harness, one family of websites. A codebase whose tools are written enum-first would have far less residue, and the interesting unpublished number is what that distribution looks like across real MCP servers.

Read from the XGrammar-2 post of 4 May 2026 and mlc-ai/xgrammar at commit 40ef651 — 22 format types in python/xgrammar/structural_tag.py, 19 registered model formats in builtin_structural_tag.py, the finite-value classifier in cpp/json_schema_converter.cc. Chart values are read off the published figures. No model was run.