Ovis-Embedding: Qwen2.5-Omni's last token as one index for text, images, video and audio

mdjsonmcpcopy↗ai

2026-09-26 · 21 min · explainer · retrieval · multimodal · omni · qwen · audio · benchmarks · open-weights

A search index that holds captions, photos, clips, podcasts and screenshots needs every one of them as a vector in the same space, with scores that mean the same thing whichever pair you compare. The usual way to get there is to bolt towers together: a text encoder, a CLIP-style image tower, an audio model aligned to one of them after the fact. Ovis-Embedding, from Alibaba's Ovis team (arXiv:2609.25165, 21 September 2026), takes the other route. It starts from a model that already reads all four modalities in one Transformer, Qwen2.5-Omni, and reads the embedding straight out of it.

The claim that travelled is "state of the art on MMEB-v3". I read the paper, the three model repos, the safetensors headers, and the MMEB leaderboard's raw per-task score files, and recomputed the benchmark myself. Every number below is labelled: reported (the paper or a model card says so), measured (I read or recomputed it from files), or reasoned (I derived it, and say how).

It is not an Ovis model

The name suggests the team's Ovis vision-language models, whose trick was a learnable visual embedding table (arXiv:2405.20797). None of that is here. All three checkpoints are Qwen models with the language-model head taken off:

• Omni-3B is initialised from Qwen2.5-Omni-3B (arXiv:2503.20215). That model is a Thinker, a causal Transformer over interleaved text, visual and audio tokens, plus a Talker that turns its states into speech. Ovis keeps the Thinker, both encoders and the tokenizer. Time-aligned multimodal RoPE (TMRoPE) gives co-occurring audio and video frames the same temporal position, so a clip and its soundtrack go in as one input.
• VL-2B and VL-9B are initialised from Qwen3.5-2B and Qwen3.5-9B: text, images and video, no audio. Their language backbone interleaves three Gated DeltaNet linear-attention layers with one full-attention layer (the recurrence I derived in the LTC and Gated DeltaNet piece).

The readout is the simplest one there is. An input, one modality or several interleaved, is formatted with a task instruction through the backbone's own chat template, run through the model once, and the final-layer hidden state at the last non-padding token is the embedding:

where $L$ is the number of layers, $\ell(x)$ the last real token, and $d$ the backbone's hidden size. With no projection head, the dimension is the backbone's: 2,048 for Omni-3B and VL-2B, 4,096 for VL-9B (reported, and matching hidden_size in each config.json, measured). In a causal model only the last token has attended to everything, which is why last-token pooling suits a decoder where mean pooling suits a bidirectional encoder. Retrieval is cosine similarity, the dot product of L2-normalised vectors.

expand

The pooling rule, for anyone wiring these checkpoints into their own pipeline. This is mine, tested on random tensors with both padding sides; the repos ship no inference code:

import torch
import torch.nn.functional as F
 
def last_token_pool(hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    """hidden: [batch, seq, d] final-layer states; attention_mask: [batch, seq], 1 = real token."""
    if bool(attention_mask[:, -1].all()):          # left-padded: last column is real in every row
        emb = hidden[:, -1]
    else:                                          # right-padded: position of the last real token
        last = attention_mask.sum(dim=1) - 1
        emb = hidden[torch.arange(hidden.size(0)), last]
    return F.normalize(emb.float(), p=2, dim=-1)   # unit length: cosine == dot product

What is in the files

I read the safetensors headers of all three repos with HTTP range requests, eight bytes for the header length and then the JSON header, without downloading a weight. Measured:

The "3B" is the Thinker's language model. The path an input actually takes through Omni-3B, backbone plus both encoders, is 4,392,299,520 parameters. The other 1,144,821,120, a fifth of an 11.07 GB bf16 download, are the LM head, the Talker and the token2wav vocoder, which the paper and the model card both say were removed. They were not removed from the files, and config.json still has enable_talker: true and the Qwen2_5OmniForConditionalGeneration architecture. Load only the Thinker, or you hold 2.29 GB of weights that never run, 1.67 GB of them speech synthesis (reasoned from the parameter counts at two bytes each). The VL checkpoints were stripped properly: VL-9B's untied head, 248,320 × 4,096 = 1.02B parameters, is gone, which is why it holds 8.39B, not nine.

One space, one loss

Every training example is a tuple: a query, one positive, and $K$ hard negatives. In a batch of $N$ tuples, all positives and all negatives are pooled into one candidate set of size $N(1+K)$, gathered across data-parallel workers, and every query is scored against all of it. At temperature $\tau$, the probability the model puts on the right answer and the per-query InfoNCE loss are:

That is the objective CLIP made standard, and the one behind the Contrastive Language Model, except that the candidates here can be any mix of text, pixels, frames and sound. Ovis changes two things about it in the first stage.

A focal weight. A query whose positive is already far ahead of its negatives contributes as much loss as one that is still confused. The paper borrows the fix from focal loss for detection (arXiv:1708.02002) and rescales each query by how unsolved it is, with the weight treated as a constant and normalised to mean one:

Here $\mathrm{sg}$ is stop-gradient and $\gamma \geq 0$ sets how hard easy queries are discounted; $\gamma = 0$ is plain InfoNCE. Because the weights average to one, the loss scale does not change. It only moves budget from solved queries to unsolved ones.

A teacher's whole ranking. InfoNCE's target is one-hot: the positive is right and every negative is equally wrong. A teacher embedding model, whose similarities over the same candidate set are precomputed offline, gives a graded target instead, and the student minimises the forward KL divergence from the teacher's softmax to its own. Stage 1 adds the two losses with equal weight "without further tuning" (reported).

The toy below is that softmax, on sixteen invented items: four concepts, each as a caption, a photo, a clip and a sound. It shows the two failure modes a shared index has, and what the training choices do to them.

The widget measures nothing. In prose: with every modality in the pool, a caption query finds its concept and then splits its probability between the photo, the clip and the sound, all right in meaning, one in the modality asked for. Give each modality a shared offset, the modality gap real encoders have, and past a point its nearest neighbours become other captions. The MMEB-v3 authors report both failures in real models. Ovis's answers are an instruction on every query, formatted through the chat template (the paper publishes no instruction set of its own and no run without one), and one-source batches, which take both failures out of the loss.

Four stages

The paper's recipe, in order (reported):

1. Low-rank contrastive pretraining. LoRA on the full mixed corpus, every modality and task in every global batch, with the focal and distillation losses above. The paper's reason for LoRA is stability: in its runs, full-parameter updates at this point "perform markedly worse", because the untrained embedding space sends large gradients into every weight and erodes what the backbone knew. No number is given for "markedly".
2. Full-parameter homogeneous finetuning. Everything is unfrozen and trained on a smaller, cleaner set with plain InfoNCE, but each micro-batch now comes from one dataset. In a mixed batch, the paper argues, other queries' candidates can be told apart by shortcuts, modality or sentence length or style, instead of by meaning. Candidates are deduplicated by hash so an in-batch copy of the positive never becomes a negative, and a step still averages gradients over micro-batches from different datasets.
3. Annealing Embedding Distillation. A stronger teacher is run over the stage-2 data; only examples the teacher gets right are kept, and those the student still misses are upsampled. The KL term's weight is set per query by the student's own confidence, $\lambda_i = \lambda_{\min} + (\lambda_{\max} - \lambda_{\min})\,\mathrm{sg}[(1-\pi_i)^{\gamma}]$, and averaged over the fixed batch size, so the teacher's pull fades as the student learns.
4. Elastic inference. A post-hoc projection to shorter vectors, covered below.

expand

The data (reported), all cast as query, positive, negatives. Images: ImageNet, CUB-200 and SUN397 plus web images labelled by Qwen3.5-Plus, TextVQA, DocVQA and InfoVQA, search pairs, COCO grounding and Quark shopping queries. Video: web clips kept only when a vision-language model, shown sampled frames, agrees they match, with the query then corrected to what is visible. Audio: event, instrument and command labels, captions and transcripts, in both directions. Text: five retrieval paradigms, domain data in law, finance, code and health, and MTEB-style tasks recast as retrieval with BGE-M3 hard negatives. Agents: tool, GUI and evidence retrieval, with BM25 negatives. The total is "approximately 50M" pairs, under a footnote calling that paragraph's numbers "placeholder estimates based on the current data freeze".

What the paper does not say. Which teacher models were distilled from; $\tau$, $\gamma$, $\lambda_{\min}$, $\lambda_{\max}$, the LoRA rank, batch sizes or step counts; or the compute, beyond "clusters equipped with NVIDIA H100 80 GB GPUs". Its evaluation section opens by promising to show "which components of the proposed recipe contribute the most", and then there is no ablation of the focal loss, homogeneous sampling, distillation or LoRA-first training. Of seven tables, the only component with numbers behind it is the elastic projection. Each stage is plausible; that each earns its place is asserted, not shown.

Shorter vectors

Matryoshka training (arXiv:2205.13147) makes prefixes of one embedding usable as shorter embeddings, but the paper reports that adding it to the multi-objective training hurt the full-width vector. So the shortening is done after training, on a frozen encoder. First a rotation: the second moments of each modality's candidate embeddings are estimated on equal-sized samples, averaged with uniform weights, and eigendecomposed once, $\Sigma = V \Lambda V^{\top}$. Rotating by the orthogonal $V$ changes no inner product; it only puts the high-variance directions first, where truncation keeps them. Then a residual linear adapter per width, initialised at zero and fitted without labels to keep each candidate's similarities to the whole batch and to its nearest neighbours as they were at full width. Both fold into one matrix per width:

expand

The paper's Table 7 (reported) holds the result. Averaged over the six suites with MMEB-v3's task counts, the adapted embedding keeps 100.1% of its full-width score at 1,024 dimensions, 99.2% at 512, 97.4% at 256 and 93.2% at 128. Cutting the same vector to its first $d$ coordinates and renormalising keeps 99.3%, 97.0%, 93.8% and 85.8%. The cost at 128 is uneven: audio drops 0.33 points and video 1.26, while visual documents drop 6.97 and agent retrieval 6.18. The paper's reading is that finding one passage on a busy page, or one changed button in a GUI, lives in low-variance directions that a short prefix throws away.

Two caveats. The 2,048 column of Table 7 averages 58.00, not the 58.46 of the headline table. The likely reason is that the adapter was fitted on the stage-2 encoder (Figure 5 labels it "frozen Stage-2 encoder"), while the model card describes the released weights as the product of all three training stages, so an adapter for them would have to be refitted (reasoned). And no adapter or projection matrix is in any of the three repos (measured, from the file listings). What you can do with the released checkpoint today is naive truncation, the 85.8%-at-128 curve.

What MMEB-v3 measures

MMEB-v3 (arXiv:2604.23321) comes from authors at the University of Waterloo, Shanghai Jiao Tong, the Eastern Institute of Technology in Ningbo and Google, among others, and extends MMEB-v2's image, video and visual-document suites with 111 new tasks. The leaderboard is a Hugging Face Space run by TIGER-Lab, fed by score files that teams upload; a file carries a data_source field, and one without it is shown as self-reported. The counts, from the leaderboard's datasets.py (measured):

The overall score is the unweighted mean over all 190 tasks, with a task a model cannot run counted as zero. So text and agent retrieval are 100 of the 190 tasks and audio is 11. An "omni-modal" number is mostly a text-and-agent number with images on the side, and the audio group, the reason to use an omni backbone at all, moves the total by at most 5.8% of its range.

The numbers, checked

I reimplemented the leaderboard's aggregation (its utils_v3.py, read, not run) over its score files at commit 8129607 of 16 September. Ovis's own file, uploaded 3 September, recomputes to 58.46 overall, and every group score in the paper's Table 1 is within 0.01 of the files (measured). Three of the four baselines are files marked "Reproduced by TIGER-Lab": e5-omni-7B, Omni-Embed-Nemotron-3B and LCO-Embedding-Omni-7B. The fourth, Tianmu-Emb-Uni, and Ovis itself are self-reported. The baselines are the leaderboard's numbers, not picks from each model's own paper, and the headline against them holds: 58.46 against Tianmu's 53.27, e5-omni-7B's 47.14, Nemotron's 43.60 and LCO's 43.14, first in all six groups (reported, and recomputed).

The files also hold models the paper did not compare against, and two of them matter (measured):

• WeMM-Embedding-9B from Tencent, a Qwen3.5-based vision-language model uploaded on 26 August, recomputes to 59.33 overall (59.55 before a correction to two of its agent tasks on 16 September). It cannot take audio, so its 11 audio tasks count as zero, and it still finishes ahead. Over the 179 non-audio tasks, it is 62.98 to Ovis's 58.97. It is the only file that beats Ovis's 58.46, and it is three times Ovis's size.
• AuroLA-Omni-7B, uploaded the same day, scores 50.49 on audio to Ovis's 50.08: lower on audio classification (71.07 to 73.30), higher on audio retrieval (33.34 to 30.73).

Both files predate the paper by nearly four weeks. "Ranks first on the aggregate score of every evaluation group" is true of the four baselines chosen, and not of the leaderboard.

Where it is weak. Audio retrieval averages 30.73 Hit@1 against 73.30 for audio classification; per task (measured) that is 21.53 on Clotho, 14.43 on AVE and 5.61 on TUTSound's hard temporal-grounding split. Putting a label on a clip works; finding the right clip from a description mostly does not. Video retrieval sits at 52.05. On text, reasoning-heavy BRIGHT is 15.66 and medical R2MED 25.68, and MultiConIR, queries with several conditions that must all hold, is 61.73, 7.94 behind Nemotron. One cosine has to fold every condition into one number; the jev-semgrep piece makes the related point that similarity cannot say "and not". Agent memory retrieval, 29.44, is second to Nemotron's 32.23.

Two slips in the paper, besides the placeholder footnote, neither of which changes the ranking (measured against its own tables):

• The text puts the audio margin at 7.04 points. That is the gap to e5-omni-7B's 43.04; the runner-up in Table 1 is LCO-Embedding-Omni-7B at 43.17, so the margin is 6.91, which is what the model card says.
• Figure 1's MMEB-v3 panel, below, replaces Tianmu, the strongest baseline in Table 1, with Qwen3-VL-Embedding-2B, whose bars read 75.0 on image and 61.9 on video. Those are its MMEB-v2 scores from the paper's own Table 6 (74.96, 61.87). The MMEB-v3 paper reports 69.5 and 55.9 for it.

expand

Beyond MMEB-v3

Audio and video suites (reported). On the beta Massive Audio Embedding Benchmark, 30 tasks, Omni-3B averages 57.29 per task against LCO-Embedding-Omni-7B's 53.54. On the beta Massive Video Embedding Benchmark, 23 tasks, it averages 61.77 against LCO-7B's 57.58. Both are the team's local runs, not leaderboard submissions; the ranks in the paper's tables were estimated by inserting them into a leaderboard snapshot. The weak spots rhyme with MMEB-v3's: audio clustering at 17.59 (the best in the table, and still low in absolute terms), audio reranking 80.70 against e5-omni-7B's 86.70, video clustering 25.35 against LCO-7B's 27.35.

Text (reported). There is no MTEB score. On RTEB's 15-task English public split, Omni-3B scores 67.35 against Qwen3-Embedding-4B's 67.27, which is a tie, not a win. Underneath, it is 49.61 to 62.67 on legal and 80.00 to 69.75 on finance, and the paper says it trained on data built for exactly RTEB's domains, law, finance, programming and healthcare, while deduplicating against the test sets. The "MMEB-Text 47.15" in the same table is MMEB-v3's own 53-task text group again, not a separate benchmark.

The VL siblings on MMEB-v2 (78 tasks, no audio). VL-9B scores 81.13, first among every file on the leaderboard I recomputed (measured). Its margin over the strongest model it was compared with, Octen-VL-Embedding-Large, is 1.04; over WeMM-Embedding-9B, which it was not compared with, 0.55. On video it trails Octen-Large, 72.90 to 75.95. VL-2B scores 77.46, and WeMM-Embedding-2B, not in its comparison, has 77.94 in the files.

Practicalities

Licence. All three repos say Apache-2.0. Qwen3.5-2B and 9B are Apache-2.0, so the VL chain is clean. Qwen2.5-Omni-3B is not: its repo carries the Qwen Research License, which grants use "FOR NON-COMMERCIAL PURPOSES ONLY" and lets derivatives carry their own terms only where use "otherwise complies" with it. Alibaba licensed the base and Alibaba released the derivative, so it may be entitled to relicense, but a README is not where that gets settled. For a product, get the Omni licence confirmed in writing, or use a VL model and give up audio (reasoned; not legal advice).

Code. None. The GitHub repos hold a README and a logo, and both READMEs still say the weights "are not open-sourced yet", though they are on Hugging Face. The model cards describe the retrieval steps in prose, with no snippet and no instruction strings beyond the Figure 2 example. An args.json in each checkpoint names a template, qwen2_5_omni_emb or qwen3_5_emb, with task_type: embedding, which reads like MS-SWIFT's format; the paper mentions MS-SWIFT when it discusses training speed (reasoned).

What an input costs, reasoned from the processor configs and the Qwen2.5-Omni report:

A 1024-pixel square image costs Omni-3B roughly 7.6 TFLOP in the Thinker (twice its 2.77B non-embedding parameters, times 1,369 tokens) plus about 7.3 in the vision encoder, which runs on the 5,476 patches before merging: two FLOPs per parameter per token, attention ignored (reasoned). A 30-token text query is about 0.17 TFLOP, so indexing images costs roughly ninety times what indexing short captions does.

Storage. A 2,048-dimensional fp16 vector is 4,096 bytes, so a million of them are 4.1 GB before any ANN structure; VL-9B's 4,096 dimensions double that. Naive truncation to 512 dimensions, keeping 97.0% of the score on Table 7's stage-2 numbers, is a quarter of the size.

What I would use it for

For an any-to-any index where audio has to sit next to everything else, this is the strongest open model in the leaderboard's files that takes audio at all, and at 3B the smallest of the top three. The mechanism is cheap to reason about: one forward pass, the last token, a cosine. Be careful where the paper is quiet: audio retrieval, unlike audio classification, is still weak; Table 1 is honest about its four baselines and silent about a stronger model that was already public; the recipe has no ablations; the checkpoint carries 1.14B parameters it never uses; and the licence chain runs through a research-only base. If you do not need audio, VL-9B has the cleaner licence and the top MMEB-v2 score, and WeMM-Embedding-9B deserves a run on your own data before you choose.