# A grammar guarantees it parses. A menu guarantees it was on the menu.

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/xgrammar-structural-tags
> date: 2026-09-22
> tags: structured-generation, constrained-decoding, agents, inference, llm, explainer
[XGrammar-2](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)
is the constrained-decoding layer under SGLang, vLLM, TensorRT-LLM and MLC-LLM, and
its headline feature is **Structural Tag** — a small JSON DSL that describes an
output shape, compiles to a grammar, and masks the model's vocabulary at every
decode step so the emitted tokens cannot leave that shape. Twenty-two format
types, nineteen built-in model formats, and one `response_format` field that
serving engines expose over the OpenAI API.

It is also the other half of an argument this site has been having with itself.

[RLCD is not constrained decoding](/articles/rlcd-calibrated-decisions) separated
two things people conflate: a *decoder-side* constraint on what a model may emit,
and a *training-side* property of how well its probabilities track reality. [What
a decision model cannot do](/articles/what-decision-models-cannot-do) then showed
what the first choice costs — a model whose output is always an element of the
caller's option set cannot write a string, and needs a generative model beside it
to do so.

XGrammar is the third position. **Do not replace the generator. Constrain it.**

So: what does each one actually promise?

<GuaranteeMatrix />

## The two guarantees, stated precisely

**A grammar guarantees membership in a language.** The masking is exact: a token
whose emission would take the prefix outside the language is assigned zero
probability, so there is no decode path to an invalid string. If your language is
"a JSON object matching this schema", you will get a JSON object matching that
schema, with probability one, forever. What you will *not* get is any promise
about which member of that language you got — a schema with one free `string`
property admits uncountably many documents and the grammar is indifferent between
all of them.

**A menu guarantees membership in a set.** The output is an element of the option
list the caller supplied on this request. It cannot be a value the caller did not
offer, cannot be a hallucinated tool name, cannot be an enum member from last
year's schema. What it will *not* give you is a value nobody enumerated, and
[the docs of the model that sells this hardest say so
plainly](/articles/what-decision-models-cannot-do): *"not trained to generate
text. While you can force it to by chaining choices, this will not work well and
will be very slow."*

Neither implies the other, and both directions of that matter.

A grammar does not imply a menu: to guarantee "the value is one of these five",
you must write the five into the grammar. Then you have a menu — which is a real
construct here, not a rhetorical one, and I will come back to it.

A menu does not imply a grammar: nothing about "choose one of these options"
guarantees the chosen option is syntactically anything. It is well-formed only if
every option you supplied was well-formed, which is a property of your code, not
of the model.

## The evidence for this is XGrammar's own benchmark

The cleanest demonstration that these come apart is in the post's first figure,
and it is the figure that should be quoted from it.

<Figure
  src="/articles/xgrammar-structural-tags/fig2.png"
  alt="Two bar charts side by side on BFCL-V3, with and without a structural tag, for five models: Llama-3.2-1B, Llama-3.2-3B, Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen2.5-72B. The left chart is schema accuracy: without the tag the values run from about 21 percent for the 1B up to 100 percent for Qwen2.5-72B; with the tag every bar is exactly 100 percent. The right chart is output accuracy: without the tag it runs from about 5 percent for the 1B to about 92 percent for Qwen2.5-72B; with the tag the 1B rises to about 46 percent, the 3B to about 88, Qwen3.6-27B to about 93, the 35B-A3B to about 90, and Qwen2.5-72B is essentially unchanged and a hair lower."
  caption="Read the two panels together. Left: schema accuracy goes to exactly 100% for every model — that is the guarantee, delivered, and there is nothing probabilistic about it. Right: output accuracy does not. The 1B goes from 5% to about 46%, which means most of the 79 points of unparseable output became parseable and wrong. (blog.mlc.ai, 'XGrammar-2: Fast and Customizable Structured Generation for Tool Calling and Agents', 4 May 2026, Figure 1, flattened onto white.)"
/>

Every output is in exactly one of three states — correct, parses but is wrong, or
does not parse — and the two panels pin all three. Decomposed:

<ParseVsRight />

**The grammar deletes the third state completely. It splits the deleted mass
between the other two, and the split is not in its control.** For Llama-3.2-1B
roughly half the recovered points became correct answers and half became tool
calls that parse cleanly and call the wrong thing. For Qwen2.5-72B, which already
conformed, the tag changed the score by a tenth of a point downward.

This is the right result and the post is honest about it: *"XGrammar is best used
to enforce format constraints, not to change the semantics of an LLM's response.
It helps downstream programs avoid fatal failures from malformed outputs."* Fatal
failures, not wrong answers. A malformed tool call throws; a well-formed wrong one
runs.

## Where the two mechanisms are literally the same construct

The interesting part of the comparison is that it has a fixed point, and it is in
XGrammar's type list rather than in anyone's argument.

```json
{ "type": "or", "elements": [
  { "type": "const_string", "value": "refund" },
  { "type": "const_string", "value": "replace" },
  { "type": "const_string", "value": "escalate" }
]}
```

That is a menu, written as a grammar. `OrFormat` over `ConstStringFormat`, both
first-class types in `python/xgrammar/structural_tag.py`. The equivalent through
JSON Schema is an `enum`, and XGrammar's C++ converter has a name for exactly this
case:

```cpp
// cpp/json_schema_converter.cc
bool TryGetFiniteValues(const picojson::object& schema, std::vector<picojson::value>* values) {
  if (schema.count("const")) { values->push_back(schema.at("const")); return true; }
  if (schema.count("enum"))  { /* ... */ return true; }
  return false;
}
// and, above the one-of classifier:
struct OneOfArmProof { enum class Kind { kTypeSet, kFiniteValues }; /* ... */ };
```

`kFiniteValues` is the grammar compiler noticing that a fragment of your schema is
a menu. When it fires, the grammar's guarantee and the decision model's guarantee
are the same guarantee, and [the token-mask arithmetic is the same
arithmetic](/articles/parallel-constrained-decoding) — restrict the vocabulary to
the legal continuations, renormalise, sample.

Which is the useful way to think about a real schema: it is a mixture. Some fields
are finite and both mechanisms bound them. Some are not, and neither does.

<ResidueLadder />

**The residue is where the two mechanisms fail in different ways and both fail.**
A grammar will emit `"query": ""` without complaint, because an empty quoted run
is a string. A decision model will not emit it, because it was not on a list —
which is not better, it is just a different failure. The 379-of-385 figure from
the [WindTunnel traces](/articles/what-decision-models-cannot-do) is the empirical
shape of that residue in a real agent: nearly every action a bounded model selected
had at least one field it could not fill.

## What Structural Tag actually adds over JSON-schema decoding

Constrained decoding against a JSON schema is old. The specific thing XGrammar-2
adds is that **the schema and the wire syntax are separated**.

<Figure
  src="/articles/xgrammar-structural-tags/fig1.png"
  alt="A left-to-right flow diagram. A user talks to an agent application, which exchanges messages with a tool-calling box above it. The agent application sends a Structural Tag — shown as a JSON document declaring a tag with begin '<answer>', content of type json_schema with status and message string properties, and end '</answer>' — to a serving engine on the right, labelled with SGLang, vLLM and TensorRT-LLM marks. Inside the serving engine, XGrammar 2 constrains the model output to follow the structure. The result flows back as a structured output, shown as the literal text '<answer>' followed by a JSON object with status success and a message, then '</answer>', and returns to the agent application."
  caption="The separation that makes this more than JSON-schema decoding: the caller declares a shape, the engine compiles it, and the model's vocabulary is masked against it. The schema in the middle describes the arguments; the begin and end markers describe the wire format, and they are independent. (blog.mlc.ai, 4 May 2026, Figure 4, flattened onto white.)"
/>

DeepSeek V4 does not emit tool calls as JSON. It emits this:

```text
<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>
```

Your tool's schema is still `{"city": {"type": "string"}}`. The surface syntax is
XML-flavoured with model-specific control tokens. Under a JSON-only constrained
decoder those are two incompatible facts; under Structural Tag they are a
`json_schema` node with a `style`:

| `style` | what the arguments look like on the wire |
|---|---|
| `json` | standard JSON |
| `qwen_xml` | `<parameter=key>value</parameter>` |
| `minimax_xml` | `<parameter name="key">value</parameter>` |
| `deepseek_xml` | `<{dsml}parameter name="key" string="true\|false">value</{dsml}parameter>` |
| `deepseek_v4_1_xml` | as above, with a space after the control token |
| `glm_xml` | `<arg_key>key</arg_key><arg_value>value</arg_value>` |
| `cohere_xml` | `<cofl:value name="key" type="raw\|json\|dict\|list">value</cofl:value>` |
| `kimi_k3_xml` | `<\|open\|>argument key="key" type="type"<\|sep\|>value<\|close\|>argument<\|sep\|>` |
| `minimax_m3_xml` | recursive namespace XML |

Nine wire formats, one schema language. That is a lot of accidental complexity to
absorb, and absorbing it in one library rather than in nine serving-engine tool
parsers is the actual contribution. The post's framing agrees: *"Supporting all of
these requires significant effort from serving engines and downstream
applications, and may still fail to match the official specification."*

The second thing it adds is composition. `TriggeredTags` lets the model write free
text until it emits a trigger string, at which point the output must follow a
structured tag — which is how you express "reason freely, then call a tool
strictly" without constraining the reasoning. That is a shape a JSON schema cannot
describe at all, and it is the shape every reasoning model now has.

## The costs, which are real but small

<Figure
  src="/articles/xgrammar-structural-tags/fig3.png"
  alt="Two charts. The left is tool-calling compilation time on a logarithmic vertical axis in milliseconds, against total tool count of 10, 100 and 500. XGrammar rises from 783.60 milliseconds to 6,697.0 to 60,308.0; XGrammar-2 rises from 45.56 to 173.33 to 731.73. The right chart is average end-to-end latency in seconds in SGLang for Llama-3.2-1B, Llama-3.2-3B and Llama-3.1-8B, with and without a structural tag. The 1B goes from about 0.265 to about 0.260 seconds, the 3B from about 0.157 up to about 0.166, and the 8B from about 0.258 to about 0.256."
  caption="Left: the 80x claim, which is 82.4x at 500 tools (60,308 ms to 731.73 ms) and 17.2x at 10. Right: per-request latency overhead, which is signed both ways — the 3B pays 5.7% and the other two save 1 to 2%. Note the qualifier in the post's own text: 'in a single-batch setting with warm-up and comparable output lengths'. (blog.mlc.ai, 4 May 2026, Figure 2, flattened onto white.)"
/>

- **Compilation.** Fifty tools of JSON schema used to be a 6.7-second stall. The
  cross-grammar cache (an automaton-based hierarchical hash that finds shared
  sub-structures — *"nearly 50% of structures end up reused"* at 50 tools) and
  repetition-state compression (`{"maxItems": 1000000}` from
  $O(n)$ to $O(1)$, 534 ms to 5.37 ms) take 500 tools from 60.3 seconds to 0.73.
  That is the 80&times;.
- **Per token.** Near zero, and signed both ways in the published bars. The mask
  comes from a precomputed token-mask cache, so the common case is a lookup.
- **Against speculative decoding.** This is the one that would bite, and it is the
  one the post spends most engineering on. A draft tree has to be masked
  node-by-node on the CPU while the GPU verifies it; done serially that is a stall
  per round. `traverse_draft_tree` walks the whole tree once and emits masks for
  every node, so the CPU work overlaps the GPU verify. Given [how much of modern
  serving is now speculative](/articles/qwen38-pd-serving), a constrained decoder
  that serialises against the draft verify is not deployable, and this is the fix.

## What this does not settle

The comparison has a limit worth stating, because the two mechanisms are not
substitutes for each other on any axis except "stop the invalid output".

**Constrained decoding does not calibrate.** Masking renormalises the softmax over
the legal subset, which is exact arithmetic and says nothing about whether the
resulting probability means anything. [The subset softmax is exact and it is not
calibrated](/articles/parallel-constrained-decoding), and that is precisely the
distinction RLCD's name collision obscures: a decoder-side constraint and a
training-side calibration property are orthogonal, and you can have either, both,
or neither.

**A decision model does not parse.** It returns an index and a probability. If you
want JSON out of it you write the JSON yourself, from the option you selected,
which is trivially correct and is the reason that architecture never has a parse
bug.

**Nothing here judges the semantics.** Both mechanisms narrow the space and
neither ranks what is left. The 46.5% output accuracy on a constrained
Llama-3.2-1B is a statement about the model, and no grammar is going to move it.

The honest composition, then, is the one the WindTunnel repository arrived at by
necessity and the one XGrammar's type list supports directly: **enumerate every
field you can, constrain the shape of the rest, and keep a list of which fields are
residue** — because those are the ones where a confidently wrong answer is
indistinguishable, to your program, from a right one.

## What I would take away

- **"100% schema accuracy" and "the model got it right" are different claims** and
  the same figure carries both. Ask which one a structured-output vendor is
  quoting.
- **A grammar's guarantee is unconditional and narrow.** Zero probability on
  invalid continuations is not a 99%; it is a proof. That is worth a great deal,
  and it is worth exactly nothing about correctness.
- **Look for the `enum`.** Every place a schema enumerates its values is a place
  the two mechanisms coincide and both guarantees apply. Every free `string` is a
  place neither does. That audit takes ten minutes and tells you where your agent
  will fail quietly.
- **Check the speculative-decoding path before adopting a constrained decoder.**
  If masking does not overlap the draft verify, constrained decoding costs you the
  speculation, which on current serving stacks is a larger number than anything the
  grammar saves.

Related reading: [why RLCD is not constrained
decoding](/articles/rlcd-calibrated-decisions); [what a decision model cannot
do](/articles/what-decision-models-cannot-do), whose WindTunnel split is the
residue measured in the wild; [parallel constrained decoding from first
principles](/articles/parallel-constrained-decoding), which builds the mask in
forty lines; and [SGLang's jump-forward decoding](/articles/sglang), which is the
same compiled-FSM machinery used for a different purpose and is currently dead
code.

<ChangeMyMind>

<Falsifier claim="Constraining Llama-3.2-1B took schema accuracy from ~21% to 100% and output accuracy from ~5% to ~46.5%, so roughly half the recovered mass became parseable-and-wrong.">
Read off the two bar charts in the post's Figure 1, to the nearest half point,
which is the weakest link in this article. The decomposition assumes output
accuracy counts a subset of schema-conforming calls — that a call scored correct
must also conform. If BFCL-V3's output-accuracy metric can credit a call that
fails schema validation (via a lenient parser, say), the three states are not
disjoint and the arithmetic is wrong. The post's evaluation scripts are linked from
it; running them would settle both the readings and the metric definition.
</Falsifier>

<Falsifier claim="A grammar and a menu coincide exactly where the grammar enumerates a finite set, and XGrammar implements that case explicitly.">
`python/xgrammar/structural_tag.py` defines `OrFormat` and `ConstStringFormat` as
first-class types; `cpp/json_schema_converter.cc` has `TryGetFiniteValues`, which
returns true for `const` and for a non-empty `enum`, and an `OneOfArmProof` whose
kinds are `kTypeSet` and `kFiniteValues`. Read at commit `40ef651`. If those are
an optimisation pass rather than the semantics — that is, if a schema `enum`
compiles to something that admits values outside the list on some path — the
equivalence is weaker than I have claimed, though a grammar could still be written
by hand to enforce it.
</Falsifier>

<Falsifier claim="Structural Tag's real contribution over JSON-schema constrained decoding is separating the schema from the wire syntax.">
`JSONSchemaFormat.style` takes nine values in `structural_tag.py`, and
`builtin_structural_tag.py` registers nineteen model formats. That is evidence of
the feature existing, not of it being the most valuable one. A reasonable person
could rank the cross-grammar cache higher — 60.3 seconds to 0.73 at 500 tools is a
deployability threshold, not a percentage — and I would not argue hard.
</Falsifier>

<Falsifier claim="A grammar guarantees the output parses but cannot make it correct, and the post agrees.">
The post's own sentence: *"XGrammar is best used to enforce format constraints,
not to change the semantics of an LLM's response."* If someone demonstrates a
grammar construction that reliably raises semantic accuracy beyond eliminating
format failures — by pruning branches that lead to known-bad continuations, say,
which is a thing a grammar could in principle encode — then the guarantee is
broader than I have drawn it. The BFCL numbers are the evidence against; they are
five models on two subsets.
</Falsifier>

<Falsifier claim="Per-token overhead is near zero, and signed both ways.">
The post's SGLang latency bars show Llama-3.2-3B slower with the tag (about 0.157
to 0.166 s) and the 1B and 8B marginally faster. The qualifier is the post's own:
*"in a single-batch setting with warm-up and comparable output lengths."* Batch
serving with cold grammars and long tool lists is a different regime and nobody
has published it. If the mask lookup contends at batch 256 the way the compile
step used to contend at 500 tools, "near-zero overhead" is a single-stream claim
only.
</Falsifier>

<Falsifier claim="The residue — fields neither mechanism bounds — is most of a real agent's tool arguments.">
This site's WindTunnel read: 385 of 413 selected actions needed a second model, and
379 of those carried at least one `string` property with no `enum` and no `const`.
One benchmark, one harness, one family of websites. A codebase whose tools are
written enum-first would have far less residue, and the interesting unpublished
number is what that distribution looks like across real MCP servers.
</Falsifier>

</ChangeMyMind>

---

Read from the XGrammar-2 post of 4 May 2026 and
[`mlc-ai/xgrammar`](https://github.com/mlc-ai/xgrammar) at commit `40ef651` — 22 format types in `python/xgrammar/structural_tag.py`, 19 registered
model formats in `builtin_structural_tag.py`, the finite-value classifier in
`cpp/json_schema_converter.cc`. Chart values are read off the published figures. No
model was run.