LFM2.5-Encoder: classification in one forward pass, zero completion tokens

The default way to classify text with an LLM today is to prompt a decoder and parse whatever comes back. Write instructions, describe the labels, ask for JSON, decode the answer one token at a time, then hand the string to a parser and hope it's valid. It works, and it's also the slow way to do something that has a much cheaper shape: a fixed-size answer picked from a fixed set of labels doesn't need to be generated at all.

That's the pitch behind Liquid AI's LFM2.5-Encoder models, released a week ago alongside a fine-tuning tutorial in their cookbook repo: one forward pass, zero completion tokens. I read the code behind that line, not just the slide it's on, and it holds up exactly as stated.

Two new encoders, built from a decoder

LFM2.5-Encoder-230M and LFM2.5-Encoder-350M are bidirectional encoders — the BERT shape, not the chat-model shape. Their parameter counts, read straight from the safetensors metadata rather than the rounded name: 229,693,184 and 354,483,968. Both use the LFM2 hybrid backbone (interleaved gated short convolutions and grouped-query attention), hidden size 1024, vocabulary 65,536, context length 8,192 tokens, and cover 15 languages. They ship under the LFM Open License v1.0, and both were created on Hugging Face on 2026-07-27 — about a week old as I write this.

The interesting part is how they were built: each size starts from the corresponding causal LFM2.5 decoder checkpoint and is converted into an encoder with three changes.

1. Bidirectional attention replaces the causal mask, so every position can see the whole sequence instead of only what came before it.
2. The short-convolution layers switch from causal padding to symmetric center padding, so a kernel mixes information from both neighbors instead of only the left one.
3. Pretraining continues with masked-language-modeling at 30% token masking — twice BERT's original 15%, which is a real methodological choice, not a rounding difference, in how hard the denoising task is made.

expand

This is the same bidirectional-vs-causal distinction the architectures gallery draws for BERT: full attention lets every position condition on the entire input, which is what you want when the job is producing a representation rather than continuing a sequence. Generation needs causal masking so the model can't peek at the future it's about to write; classification has no future to hide — the whole document is already there, and every token should get to see all of it before you summarize what the document means.

The mechanism: mean-pool, one Linear, sigmoid

Liquid's fine-tuning tutorial (examples/lfm-encoder-classification/ in the cookbook) is a complete, runnable project — train.py, predict.py, a YAML config, sample data — and the model class inside it, DocumentClassifier, is short enough to read in full. It loads the pretrained encoder, throws away the masked-token prediction head it was pretrained with, and keeps only the backbone:

outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
hidden = outputs.last_hidden_state              # [batch, seq_len, 1024]
 
mask = attention_mask.unsqueeze(-1).to(hidden.dtype)
pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp_min(1.0)
 
logits = self.classifier(self.dropout(pooled))   # one nn.Linear(1024, num_labels)
loss = nn.functional.binary_cross_entropy_with_logits(logits, labels.float())

That's the entire model on top of the backbone: an attention-masked mean over the last hidden state,

(padding tokens carry $m_i = 0$ so they don't dilute the average), then one nn.Linear sized [hidden, num_labels], trained with binary_cross_entropy_with_logits — the loss for multi-label classification, where a document can carry zero, one, or several labels, as opposed to cross-entropy's "exactly one correct class." At inference, predict.py does the other half in three lines:

with torch.inference_mode():
    probabilities = torch.sigmoid(model(**inputs).logits)[0].cpu().tolist()
predicted = [label for label, p, t in zip(labels, probabilities, thresholds) if p >= t]

No generate(), no sampling, no stop tokens, no text to hand to a parser. One forward pass in, a fixed-length array of probabilities out, compared against per-label thresholds tuned on a validation split. That is what "zero completion tokens" means in the code, not just the slide.

Sigmoid, not softmax — and why it has to be

The reason this needs its own head, rather than reusing whatever classifier head ships with a decoder fine-tune, is the difference between picking one thing and scoring several independently. Softmax turns a set of logits into a probability distribution that sums to 1 — it is built to choose exactly one winner, which is correct for single-label problems (a document is sports, politics, or tech, never more than one). Sigmoid applied per label makes each label its own independent yes/no question: $\sigma(z_i) = 1/(1+e^{-z_i})$, with no normalization across labels, so two labels can both clear the threshold, or none can. A support ticket about a double charge is legitimately both a billing issue and a technical one; softmax would be forced to pick a single "real" answer and quietly discard the other.

Where it lands: 4th of 14, honestly

Liquid's own eval — a 17-task suite spanning GLUE, SuperGLUE, and five multilingual tasks, averaged over 5 seeds with standard deviations reported — puts LFM2.5-Encoder-350M 4th of 14 models at a mean score of 81.02, and LFM2.5-Encoder-230M 6th at 79.29.

That ranking is worth sitting with rather than rounding up. LFM2.5-Encoder-350M sits a point below ModernBERT-large and two points below XLM-R XL — a model ten times its size. It is not the best encoder on this benchmark, and Liquid doesn't present it as one. It's the fourth-best, at a fraction of the parameters of the model above it, next to a considerably larger one. The 230M model separately beats ModernBERT-base despite being the larger of the two by parameter count — a real, checkable comparison the raw numbers support either way you read them.

expand

The chart also settles a question Liquid answers candidly in the same blog post: why build a new general-purpose encoder instead of reusing their existing retrieval models? Because those retrieval-tuned siblings — LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M — score 76.18 and 75.68 on this same suite, both below the general-purpose LFM2.5-Encoder-350M's 81.02. In their own words: "Because retrieval is only a subset of what encoders enable, we chose to build a general-purpose encoder rather than adapt the existing retrievers." A model tuned to make embeddings cluster well for search is not automatically a good classification backbone, and Liquid's own numbers show the gap rather than hiding it.

On raw speed, Liquid also reports the 350M encoder running about 3.3× faster than ModernBERT-base at 8,192 tokens on CPU; a separate blog claim puts the 230M model specifically at roughly 3.7× faster than ModernBERT-base at the same length (about 28 seconds versus over a minute and a half). Those are two distinct comparisons, not one number restated — worth keeping straight if you quote either.

The tutorial's own result

The cookbook ships a second, harder example beyond the 4-label sample data: fine-tuning LFM2.5-Encoder-350M on ECtHR-A (coastalcph/lex_glue), European Court of Human Rights cases labeled by which of 10 Convention articles they violate — real long documents, real multi-label targets, 9,000 / 1,000 / 1,000 train/validation/test examples, CC BY 4.0. Trained at the full 8,192-token context, one seed, 3 epochs:

The README says outright that these numbers are from one seed — no variance reported, unlike the 17-task pretraining eval above. Read it as "this recipe works on a real long-document benchmark," not as a tuned, reproducible leaderboard number. Thresholds are tuned only on the validation split and never touch test until a separate, explicit --evaluate-test flag is passed — a small detail, but the right one for anyone checking the tutorial's methodology.

What you give up

An encoder with a classification head cannot do several things a decoder can, and it's worth naming them plainly rather than only listing what it's good at:

The honesty gaps worth naming too: these models are about a week old, with limited independent adoption to point to yet. The 17-task eval is Liquid's own compilation — methodologically solid (5 seeds, reported std, a full 14-model field rather than a curated subset), but not yet replicated by anyone outside Liquid. And the cookbook repo carries no LICENSE file at its root as of this writing, which matters if you plan to reuse the tutorial code itself, distinct from the separately-licensed model weights.

The take

The argument here isn't that encoders are back or that decoders are wrong for classification — it's narrower and more useful than that: match the tool to the shape of the answer. If you already know the output is a choice from a fixed set of labels, a bidirectional encoder can produce that choice as a probability vector in one forward pass, with nothing to decode and nothing to parse. If you don't know the shape of the answer in advance — you need explanation, planning, or free text — that's what the decode loop and its own cost structure are for. LFM2.5-Encoder is a clean, current example of the first case done right: a small model, an honestly-reported 4th-of-14 ranking against models many times its size, and a fine-tuning recipe short enough to read start to finish in one sitting.

Built on Liquid AI's LFM2.5-Encoders blog post, the LiquidAI/LFM2.5-Encoder-230M and LiquidAI/LFM2.5-Encoder-350M model cards, and the examples/lfm-encoder-classification tutorial in Liquid4All/cookbook. Parameter counts are read from HF safetensors metadata, not the rounded model names. The 17-task benchmark and both embedded figures are Liquid AI's own, reproduced for commentary; the two interactive diagrams are original illustrations of the mechanism using illustrative example data, not measured traces.