GLiNER2.5-Decide: the options are tokens too

mdjsonmcpcopy↗ai

2026-09-26 · 23 min · explainer · encoders · small-models · calibration · on-device · benchmarks

"Introducing GLiNER2.5-Decide, our new 340M parameter open weight, encoder-based decision model." — Fastino, 24 September

This site has spent two weeks on decision models: hand one a question and a closed list of answers, get back one answer and a probability, never a sentence. Most of them have been decoders with the vocabulary head sliced off or replaced — Jev, Kev, AgentJev, Jev-Omni — plus one scorer trained from scratch, CUA-S1. Laya was the one encoder. GLiNER2.5-Decide is the second, from the family that deleted the width axis a month ago, and a clean enough design to take apart properly.

Three things to answer. How does an encoder choose among options it has never seen, without generating the answer? What are the probabilities it returns? And do the claims hold — the 340M, the benchmark lead over Jev-style decoders, and FluidInference's Core ML port at "~4× faster, ~5× less peak RAM, half the size"?

What 340M counts

A safetensors file starts with an eight-byte length and a JSON header listing every tensor's name, dtype and shape. Two range requests give the whole parameter table without downloading 1.9 GB:

The decision path is 436,022,273 parameters. FluidInference's conversion report arrives at exactly that figure for what it exported — an independent check that this is the partition the code uses.

None of the obvious readings gives 340M. The whole file is 486.4M; the decision path is 436.0M; everything except the embedding table is 355.4M. The number comes from the base model: fastino/gliner2-large-v1's card says "Parameters: 340M" for a checkpoint whose own safetensors total is also 486,444,053 — the same 419 tensors, with the same names, shapes and dtypes. Decide inherited the label along with the weights. The multilingual sibling's "287M" is, by contrast, exactly its file (287,355,159).

The config is the more interesting read:

// fastino/GLiNER2.5-Decide · config.json
"architecture": "span",
"architectures": ["SpanExtractor"],
"max_width": 8,
"model_name": "microsoft/deberta-v3-large"

This is a GLiNER2 span model: the architecture the GLiNER2.5 piece described as enumerating every start with every width up to a ceiling of eight. GLiNER2.5-multi-Decide is the one built on the boundary architecture ("architecture": "boundary", max_len: 4096, mDeBERTa-v3-base), and it scores 56.7% to the English model's 60.2%. So "2.5" names the family and the library (gliner2 2.0.0), not the architecture. For decisions that changes nothing — the span head never runs. For the blog's line that the model "can extract spans and relations", it means the width-8 ceiling is back.

How an encoder makes a decision

A decoder decision model reads its answer off the next-token distribution or a head on a final hidden state. An encoder has no "next". GLiNER's answer, from the GLiNER2 paper and unchanged here, is to make the labels part of the input.

classify_text(text, {"intent": ["refund_request", "cancel_subscription", "login_problem"]}) becomes one flat sequence. The processor (_transform_schema, then _format_input_with_mapping) writes:

( [P] intent ( [L] refund_request [L] cancel_subscription [L] login_problem ) ) [SEP_TEXT] my subscription renewed on april 15 …

[P], [L] and [SEP_TEXT] are added special tokens. A second head is appended after a [SEP_STRUCT], before [SEP_TEXT]. A label description goes into the prompt string after a [DESCRIPTION] token; a question goes after the head name (answer: Did the treaty enter into force in 1992?). The text is split into words and lower-cased word by word before it is tokenised, so URGENT and urgent are the same input.

DeBERTa-v3-large reads the whole thing bidirectionally in one pass. Then, for each head, the scorer takes the final hidden state at every [L] marker and pushes it through the classifier:

where $h_{[L]_k}\in\mathbb{R}^{1024}$ is the marker state for label $k$, $W_1$ is 2048×1024, and $w_2$ is a 2048-vector. One scalar per label, a softmax across the head for single-label tasks, an independent sigmoid per label against cls_threshold for multi-label ones. There is no vocabulary and no decoding step, so there is nothing to parse and no way to return a label that was not offered.

That is the whole design, and it cuts both ways.

What it buys. The option set is data. The classifier's output dimension is 1, so two labels or twenty-eight cost the same weights, and a label with a description is just a longer string in the same slot. The marker is not a fixed embedding compared against a document vector; it is a token that has read the text, and the text has read it. That is how one checkpoint does intent, sentiment, a yes/no gate and a 0–10 scale without any head changing shape.

What it costs. Nothing about the arrangement isolates an option. Every [L] reads every other label, and DeBERTa's disentangled attention is relative-position: reverse the list and each marker sits at a different distance from the text and from its neighbours. The per-option scorer is invariant by construction because each option is scored alone at the same position; Kev is not, because its options share one causal sequence. GLiNER is further along the same axis than Kev: its sequence is bidirectional, so an early label reads a later one too, and so does the text.

The upside of that, and it is a real one, is that a question where one option's content decides another is expressible here. Jev scored 0 of 100 on exactly that kind of question because its options never share a context. GLiNER's always do.

expand

Order is part of the input

GLiNER's training code knows this. SamplingConfig in the public gliner2 package defaults to shuffle_classification_labels=True, drops up to half of a head's labels at random, and with probability 0.5 replaces the real label names with label 1, label 2, … and moves the meaning into descriptions. That is data augmentation aimed squarely at making the logits depend on what a label says rather than where it sits. It is also an admission that position is visible to the model. (Whether Decide's fine-tune kept these defaults is not published.) ClassificationScorer's docstring even notes that it recovers label order from the tokens that were actually encoded, because the processor reorders them.

Training toward invariance is not the same thing as being invariant, so I measured it. Fastino runs a public playground Space for this model — fastino/gliner25-decide-playground, CPU, the official gliner2 Classifier with the independent decoder — and its /api/infer endpoint returns the full probability vector for every head. I took the first ten rows of each of the 17 domains in the public Fast Decisions development split and, for each row, the first single-label head: 170 decisions. Each went in three times — labels as the dataset gives them, reversed, and one seeded shuffle — for 340 reordered comparisons.

Eleven of 340 reorderings changed the answer, spread over nine questions. That puts GLiNER2.5-Decide about where Kev's 9B checkpoint sits on its own ladder — 1.8% flips and 0.0326 mean movement there, 3.2% and 0.0313 here — and above hosted Jev's 0.0202. Not a large number, and not zero.

The distribution matters more than the mean. The median comparison moved by 0.005, 100 of the 340 moved by nothing the endpoint can show, and 29 moved by 0.1 or more. Every one of the nine questions that flipped had its top label at 0.70 or below in the original order: travel requests choosing between visa_docs, hotel and other; clinic messages between referral, insurance and provider_message; an email filed under support or security. Order sensitivity lives where the model is already unsure. That is the least harmful place for it, and exactly the place anyone thresholding the confidence is looking.

Two cautions. The playground rounds to three decimals, so movement below 0.001 is invisible — irrelevant at a mean of 0.0313, decisive at the zero end of the Kev ladder. And 170 is a sample; I did not want to put 5,100 requests through somebody's free-tier Space.

So are the other heads

The model card sells multi-head calls: "One call scores all three heads on the same text, so the router does not run the model three times." True, because every head's labels are in the same sequence. Which makes every head's labels context for every other head's.

The card's own example shows it. It is a compliance email, three heads, and the card prints {"intent": "request", "urgency": "high", "route": "legal"} as the "potential output".

Asked alone, urgency comes back high at 0.386. Add the intent head: still high, 0.398. Add the route head as well — the call exactly as the card writes it — and urgency comes back critical at 0.412, with high at 0.346. Nothing about the email changed. A route head with six department names changed which urgency the model picked. FluidInference's conversion report, run on the same weights locally in PyTorch, records the same three-head result to the third decimal (0.4121) — and the weights it converted carry the same LFS hash as today's — so this is the checkpoint, not the playground.

It is not a one-off. For 30 rows from the three domains with three or more heads — email_triage, sports_recap, ticket_route — I scored every head alone and then all of them in one call.

Five answers in a hundred changed because other questions were asked in the same call, and the probabilities moved more than twice as far as they did under reordering. The changes are the kind a router cares about: is_phishing went from no to yes, an email's category from support to security, a sports recap's upset from no to yes. It is not obviously a loss — the joint calls got 62 of the 100 heads right, the single-head calls 60 — it is a change. A router that scores intent in one call on Monday and intent plus urgency plus route on Tuesday is running two different classifiers.

In the card's example, the probability of critical climbs from 0.334 alone to 0.349 with the intent head and 0.412 with both, and the route head is what tips it over.

There is a quieter consequence for the benchmark. The Fast Decisions card's scoring loop calls classify_text once per head — {head["task"]: head["labels"]} — and FluidInference followed that protocol. So the published accuracy is the accuracy of single-head calls, and the multi-head mode the model card advertises is not what was scored.

What the probabilities are

The blog says the model returns "valid answers, probabilities, confidence scores". Here is what those are in gliner2 2.0.0.

• Single-label heads: a softmax over the classifier logits at temperature 1.0. The checkpoint ships no fitted temperature; temperature is a per-task argument that defaults to 1.0.
• Multi-label heads: an independent sigmoid per label, kept if it clears cls_threshold (default 0.5; the card uses 0.4). If nothing clears it, the single best label is returned anyway.
• Training: the public training code uses binary_cross_entropy_with_logits per label, so single-label serving takes a softmax over logits that objective trains as independent sigmoids. Whether Decide's fine-tune used it is not stated.
• Calibration tooling exists, unwired. joint_ie/calibration.py has TemperatureCalibrator, fit_binary_temperature and expected_calibration_error; ClassificationConfig has a calibrator field that nothing in the classification path reads.

Fastino's own agent skill, shipped in the model repo, says the careful thing: "Do not assume confidence scores form a normalized probability distribution or convert scores between sigmoid and softmax without a supported contract." No ECE, Brier score or reliability diagram is published for Decide. On the 170 single-head decisions above, as given:

An ECE on 170 decisions is coarse, but the direction is not in doubt: the model is overconfident. It chose a label at 0.9 or above on 101 of the 170 and was right on 84.2% of those, at a mean stated confidence of 0.985. The 31 decisions below 0.6 were right less than a third of the time, so the ranking of confidences carries information even where the values do not. That is the usual shape of an uncalibrated softmax, and the usual fix — a temperature fitted on held-out data — is a few lines with tools already in the package.

The card's laptop review is the single-example version. The card prints mixed; the model says positive at 0.998, with mixed at 0.001 (FluidInference's local run: 0.9995). Take positive off the list and it answers neutral at 0.594 with mixed at 0.301. Either answer is arguably wrong. What 0.998 is not is the probability that it is right.

There is one more place the word "probability" needs care: joint decoding. Fastino's guardrail figure shows a rule turning a 0.52 safe into an unsafe at 0.87 and a 0.82 prompt-injection score into 0.94.

expand

For a guardrail that matters. In the released library a rule changes which label is chosen; the probability next to it is still the head's own view, which, whenever the rule did anything, is the view the rule overrode. Read feasible and the violations list, not the confidence.

The benchmark

Fast Decisions is Fastino's own suite: 17 operational domains, 300 held-out test rows each (5,100), plus a public development split of 100 rows each — 1,700 rows, 2,900 heads. Every head has a fixed label inventory from 2 labels (agent_handoff) to 28 (support_intent); three heads are multi-label and scored by exact set match. The score is the mean of the 17 domain accuracies. The blog calls it "unseen" and "internally generated"; the dataset card calls it "the classification suite behind GLiNER2.5-Decide". Both can be true, and the second is the one to keep in mind: the same team generated the data and trained the model.

expand

The blog is explicit about what most readers will get wrong: "This is an internal benchmark, not JevBench, and JevK5 is an open reproduction rather than TypeSafe's Jev." What the table shows is a 0.44B-on-the-path encoder ahead of two 4B decoder reproductions by 2.6 and 3.8 points on its authors' own suite, and ahead of the two encoder alternatives by 11.2 and 13.6 points. Respectable, and narrow.

Two small inconsistencies. The blog says 60.1% and 57.5%; both cards say 60.2% and 57.6%. And the 59.6% row is "GLiNER2.5-Decide-1B" on the model card but "GLiNER2 XL (1B)" on the dataset card. fastino/GLiNER2.5-Decide-1B is a separate 1,188,796,693-parameter checkpoint on a 1B Ettin encoder, so either the fine-tuned 1B model scores below the smaller one, or the row is its base. Neither card says which.

The test split is private, so nobody outside Fastino can reproduce 60.2%. The development split is public, and FluidInference scored the native checkpoint on it: 62.93% mean of domain accuracies, 61.38% pooled over 2,900 heads. Where the blog quotes test-split domains, dev agrees (support intent 75.3% vs 76%; banking 64.3% vs 64%). What 63% means depends on the floor under it, which nobody has published, so I computed it from the same 1,700 rows.

A uniform guess on every head averages 20.2% across the domains; answering every head with its most common gold label averages 25.8%. So 62.93% is about 37 points above a model that reads nothing, and most of that comes from the many-label heads. The margins are not even. The thinnest are in the domains that mix binary heads with multi-label exact-set scoring: screen_tags is 10 points above its majority baseline, restaurant_review 18, ticket_route 18.3, email_triage 20.75, and agent_handoff — a single yes/no gate, 76% against 54% — 22. Yes/no gates are where the card places a lot of its examples: handoff, "did the agent finish", spam, a question over a passage. They are also where a model gets half its accuracy for free.

No independent harness has scored Decide yet. DecisionBench, whose results repository already carries SemIf, has an open intake issue for it; the first thing it asks is whether the model exposes a score for every supplied candidate, which the Classifier API does.

Latency is Fastino's: 167.3 ms p50 on a 48-vCPU Xeon Platinum 8581C, 38.3 ms on a V100, both at batch 1 and 64 tokens with a two-head, 15-label schema. On the free-tier playground (2 vCPUs), the median server-side time over my 240 single-head requests was 1,336.5 ms. Those requests run to a median of 177 subword tokens with the schema included, against their 64-token document, on a twenty-fourth of the cores, so it is not a check on their number — only a sense of what a small CPU does with it.

The Core ML port

FluidInference's gliner2-5-decide-coreml, published the same day, is the most carefully documented conversion I have read in this series. It exports only the classification path — encoder plus classifier, the 436,022,273 parameters above — into fixed-shape packages: 128, 256 or 512 tokens, up to 4 heads of up to 32 labels. The host tokenises with the upstream processor and passes the position of every [L] marker; the package returns logits.

The "~4× faster, ~5× less peak RAM, half the size" comes from their FluidUse 0.3.0 release, and decomposes cleanly:

All four hold as stated. The thing to be precise about is the comparison: it is Core ML against PyTorch on MPS, on a Mac, not against a CPU. Their own latency report shows where Core ML runs this graph. With compute units ALL, the 256-token fp16 package takes 14.7 ms p50 for a three-head request; with CPU_AND_NE it takes 689 ms. The Neural Engine is not what makes it fast. The GPU is.

What they checked, and published:

• The export wrapper matches native logits to 4.8e-7; the packages match native classify_text on the card's three examples (that report is the independent critical 0.412 above).
• On the development split at 512 tokens: native 62.93%, fp16 62.93% with 0 heads changed, w8 63.01% with 7 changed, lut6 63.03% with 42 changed. A per-tensor 4-bit LUT was tried and rejected: 62.26%, 236 heads changed, −4 points on paper_field.
• W8 and LUT6 shrink the download (477 MB and 361 MB) and do not make it faster. In the Swift app, W8 peaked at 8.7 GB while loading, which is why the app ships fp16.

And the one constraint that matters most for anyone building on it: the buckets are fixed. FluidInference reports that the 128-token bucket fits 10.7% of Fast Decisions heads, the 256-token one 95.0%, and only 512 all of them. I reproduced that with the official gliner2 processor — tokeniser only, no weights — on the 2,900 single-head requests: 310 fit in 128 subword tokens (261 if the head also has to fit the 8-label package), 2,754 in 256, all of them in 512. The median request is 177 subword tokens with the schema included; support_intent's 28 labels push its median to 259, and the longest is 431.

The Python runtime raises ValueError on a request that does not fit. The Swift API exposes .decide at 128 tokens and .decideLong at 256, so the shorter variant in the app accepts about one benchmark-style request in ten. For a router whose inputs are one-line chat messages, 128 is fine. For a paragraph of email plus a schema, it is not.

Where it fits

Against the four gates this site uses for decision models: a finite answer set known before the call (yes, it is the schema); one your code can enumerate (yes); options judgeable without reading each other (no — GLiNER's always read each other, which makes relational questions possible and order not free); and a probability worth branching on (not established — nothing is calibrated, and under a rule the confidence is the pre-rule one).

Concretely: use it where the label set is fixed per deployment — a support queue's intents, a document-type gate in front of extraction, a moderation policy. Freeze the schema — label order and the set of heads per call — as part of the model version, because both are inputs. Pick a bucket that fits your real inputs, not the demo's. Calibrate before you threshold; with a held-out set and fit_binary_temperature per head, that is an afternoon. And keep it on English text under 512 subword tokens, schema included: that is the backbone's position configuration, and it covers everything the model was scored on.

The ledger

Genuinely new. An open, Apache-2.0 encoder decision model whose option set is data at call time, with a published development split and a scoring protocol you can run. A classification engine in gliner2 2.0.0 with constraints, feasibility reporting and exact or beam search over joint assignments. A Core ML port with agreement reports, rejected variants and bucket coverage published alongside it.

Overstated. "340M": the file is 486.4M and the decision path 436.0M. "GLiNER2.5": the English checkpoint is a GLiNER2 span model; the boundary architecture is in the multilingual one. The card's "potential outputs": two of the examples I ran return something else, one of them because the card's own multi-head call changes the answer. The joint-decoding figure's probabilities, which the library does not produce.

Unmeasured. Calibration, by Fastino. Order and head sensitivity, by anyone before this. A score on a benchmark Fastino did not build. Jev itself. And the multi-head mode, which the card sells and the benchmark protocol never calls.

The design is good, and the reason it is good is the same as the reason for every caveat above. Putting the options in the sequence is what lets a 24-layer encoder make any decision you can name without a new head. It is also what makes the order, the neighbours and the list itself part of what the model reads. The options are tokens too.