# Ovis-Embedding: Qwen2.5-Omni's last token as one index for text, images, video and audio
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/ovis-embedding
> date: 2026-09-26
> tags: explainer, retrieval, multimodal, omni, qwen, audio, benchmarks, open-weights
A search index that holds captions, photos, clips, podcasts and screenshots needs every one of
them as a vector in the same space, with scores that mean the same thing whichever pair you
compare. The usual way to get there is to bolt towers together: a text encoder, a CLIP-style
image tower, an audio model aligned to one of them after the fact. Ovis-Embedding, from
Alibaba's Ovis team ([arXiv:2609.25165](https://arxiv.org/abs/2609.25165), 21 September 2026),
takes the other route. It starts from a model that already reads all four modalities in one
Transformer, Qwen2.5-Omni, and reads the embedding straight out of it.
The claim that travelled is "state of the art on MMEB-v3". I read the paper, the three model
repos, the safetensors headers, and the MMEB leaderboard's raw per-task score files, and
recomputed the benchmark myself. Every number below is labelled: **reported** (the paper or a
model card says so), **measured** (I read or recomputed it from files), or **reasoned** (I
derived it, and say how).
| | |
|---|---|
| Paper | [arXiv:2609.25165](https://arxiv.org/abs/2609.25165), "Ovis-Embedding Team", Alibaba Token Hub |
| Weights | [ATH-MaaS/Ovis-Omni-Embedding-3B](https://huggingface.co/ATH-MaaS/Ovis-Omni-Embedding-3B), [Ovis-VL-Embedding-2B](https://huggingface.co/ATH-MaaS/Ovis-VL-Embedding-2B), [Ovis-VL-Embedding-9B](https://huggingface.co/ATH-MaaS/Ovis-VL-Embedding-9B), all labelled Apache-2.0 |
| Code | [ATH-MaaS/Ovis-Omni-Embedding](https://github.com/ATH-MaaS/Ovis-Omni-Embedding) and [ATH-MaaS/Ovis-VL-Embedding](https://github.com/ATH-MaaS/Ovis-VL-Embedding): a README and a logo each |
| Benchmark | [MMEB-v3](https://arxiv.org/abs/2604.23321), 190 tasks, leaderboard at `TIGER-Lab/MMEB-Leaderboard` |
## It is not an Ovis model
The name suggests the team's Ovis vision-language models, whose trick was a learnable visual
embedding table ([arXiv:2405.20797](https://arxiv.org/abs/2405.20797)). None of that is here.
All three checkpoints are Qwen models with the language-model head taken off:
- **Omni-3B** is initialised from Qwen2.5-Omni-3B ([arXiv:2503.20215](https://arxiv.org/abs/2503.20215)).
That model is a Thinker, a causal Transformer over interleaved text, visual and audio tokens,
plus a Talker that turns its states into speech. Ovis keeps the Thinker, both encoders and the
tokenizer. Time-aligned multimodal RoPE (TMRoPE) gives co-occurring audio and video frames the
same temporal position, so a clip and its soundtrack go in as one input.
- **VL-2B** and **VL-9B** are initialised from Qwen3.5-2B and Qwen3.5-9B: text, images and video,
no audio. Their language backbone interleaves three Gated DeltaNet linear-attention layers
with one full-attention layer (the recurrence I derived in
[the LTC and Gated DeltaNet piece](/articles/ltc-gated-delta)).
The readout is the simplest one there is. An input, one modality or several interleaved, is
formatted with a task instruction through the backbone's own chat template, run through the
model once, and the final-layer hidden state at the last non-padding token is the embedding:
$$
\mathbf{e}(x) = \mathbf{h}^{(L)}_{\ell(x)} \in \mathbb{R}^{d}
$$
where $L$ is the number of layers, $\ell(x)$ the last real token, and $d$ the backbone's hidden
size. With no projection head, the dimension is the backbone's: **2,048** for Omni-3B and VL-2B,
**4,096** for VL-9B (reported, and matching `hidden_size` in each `config.json`, measured). In a
causal model only the last token has attended to everything, which is why last-token pooling
suits a decoder where [mean pooling suits a bidirectional encoder](/articles/lfm2-5-encoders).
Retrieval is cosine similarity, the dot product of L2-normalised vectors.
The pooling rule, for anyone wiring these checkpoints into their own pipeline. This is mine,
tested on random tensors with both padding sides; the repos ship no inference code:
```python
import torch
import torch.nn.functional as F
def last_token_pool(hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
"""hidden: [batch, seq, d] final-layer states; attention_mask: [batch, seq], 1 = real token."""
if bool(attention_mask[:, -1].all()): # left-padded: last column is real in every row
emb = hidden[:, -1]
else: # right-padded: position of the last real token
last = attention_mask.sum(dim=1) - 1
emb = hidden[torch.arange(hidden.size(0)), last]
return F.normalize(emb.float(), p=2, dim=-1) # unit length: cosine == dot product
```
### What is in the files
I read the safetensors headers of all three repos with HTTP range requests, eight bytes for the
header length and then the JSON header, without downloading a weight. Measured:
| Component | Omni-3B | VL-2B | VL-9B |
|---|---:|---:|---:|
| Language backbone | 3,085,938,688 | 1,881,825,088 | 7,936,684,544 |
| Vision encoder | 668,684,288 | 331,416,576 | 456,010,480 |
| Audio encoder | 637,676,544 | — | — |
| LM head (unused) | 311,164,928 | tied | — |
| Talker (unused) | 384,604,928 | — | — |
| token2wav vocoder (unused) | 449,051,264 | — | — |
| **Total in the checkpoint** | **5,537,120,640** | **2,213,241,664** | **8,392,695,024** |
The "3B" is the Thinker's language model. The path an input actually takes through Omni-3B,
backbone plus both encoders, is **4,392,299,520** parameters. The other **1,144,821,120**, a
fifth of an 11.07 GB bf16 download, are the LM head, the Talker and the token2wav vocoder, which
the paper and the model card both say were removed. They were not removed from the files, and
`config.json` still has `enable_talker: true` and the `Qwen2_5OmniForConditionalGeneration`
architecture. Load only the Thinker, or you hold 2.29 GB of weights that never run, 1.67 GB of
them speech synthesis (reasoned from the parameter counts at two bytes each). The VL checkpoints
were stripped properly: VL-9B's untied head, 248,320 × 4,096 = 1.02B parameters, is gone, which
is why it holds 8.39B, not nine.
## One space, one loss
Every training example is a tuple: a query, one positive, and $K$ hard negatives. In a batch of
$N$ tuples, all positives and all negatives are pooled into one candidate set of size
$N(1+K)$, gathered across data-parallel workers, and every query is scored against all of it. At
temperature $\tau$, the probability the model puts on the right answer and the per-query InfoNCE
loss are:
$$
\pi_i = \frac{e^{\,\mathrm{sim}(x_i,\,y_i^{+})/\tau}}{\sum_{c \in \mathcal{C}} e^{\,\mathrm{sim}(x_i,\,c)/\tau}},
\qquad \ell_i = -\log \pi_i
$$
That is the objective CLIP made standard, and the one behind
[the Contrastive Language Model](/articles/contrastive-language-model), except that the
candidates here can be any mix of text, pixels, frames and sound. Ovis changes two things about
it in the first stage.
**A focal weight.** A query whose positive is already far ahead of its negatives contributes as
much loss as one that is still confused. The paper borrows the fix from focal loss for detection
([arXiv:1708.02002](https://arxiv.org/abs/1708.02002)) and rescales each query by how unsolved
it is, with the weight treated as a constant and normalised to mean one:
$$
a_i = \mathrm{sg}\big[(1-\pi_i)^{\gamma}\big],
\qquad
\mathcal{L}_{\text{focal}} = -\frac{\sum_i a_i \log \pi_i}{\sum_i a_i}
$$
Here $\mathrm{sg}$ is stop-gradient and $\gamma \geq 0$ sets how hard easy queries are
discounted; $\gamma = 0$ is plain InfoNCE. Because the weights average to one, the loss scale
does not change. It only moves budget from solved queries to unsolved ones.
**A teacher's whole ranking.** InfoNCE's target is one-hot: the positive is right and every
negative is equally wrong. A teacher embedding model, whose similarities over the same candidate
set are precomputed offline, gives a graded target instead, and the student minimises the
forward KL divergence from the teacher's softmax to its own. Stage 1 adds the two losses with
equal weight "without further tuning" (reported).
The toy below is that softmax, on sixteen invented items: four concepts, each as a caption, a
photo, a clip and a sound. It shows the two failure modes a shared index has, and what the
training choices do to them.
The widget measures nothing. In prose: with every modality in the pool, a caption query finds
its concept and then splits its probability between the photo, the clip and the sound, all right
in meaning, one in the modality asked for. Give each modality a shared offset, the modality gap
real encoders have, and past a point its nearest neighbours become other captions. The MMEB-v3
authors report both failures in real models. Ovis's answers are an instruction on every query,
formatted through the chat template (the paper publishes no instruction set of its own and no
run without one), and one-source batches, which take both failures out of the loss.
## Four stages
The paper's recipe, in order (reported):
1. **Low-rank contrastive pretraining.** LoRA on the full mixed corpus, every modality and task
in every global batch, with the focal and distillation losses above. The paper's reason for
LoRA is stability: in its runs, full-parameter updates at this point "perform markedly worse",
because the untrained embedding space sends large gradients into every weight and erodes what
the backbone knew. No number is given for "markedly".
2. **Full-parameter homogeneous finetuning.** Everything is unfrozen and trained on a smaller,
cleaner set with plain InfoNCE, but each micro-batch now comes from one dataset. In a mixed
batch, the paper argues, other queries' candidates can be told apart by shortcuts, modality or
sentence length or style, instead of by meaning. Candidates are deduplicated by hash so an
in-batch copy of the positive never becomes a negative, and a step still averages gradients
over micro-batches from different datasets.
3. **Annealing Embedding Distillation.** A stronger teacher is run over the stage-2 data; only
examples the teacher gets right are kept, and those the student still misses are upsampled.
The KL term's weight is set per query by the student's own confidence,
$\lambda_i = \lambda_{\min} + (\lambda_{\max} - \lambda_{\min})\,\mathrm{sg}[(1-\pi_i)^{\gamma}]$,
and averaged over the fixed batch size, so the teacher's pull fades as the student learns.
4. **Elastic inference.** A post-hoc projection to shorter vectors, covered below.
**The data** (reported), all cast as query, positive, negatives. Images: ImageNet, CUB-200 and
SUN397 plus web images labelled by Qwen3.5-Plus, TextVQA, DocVQA and InfoVQA, search pairs, COCO
grounding and Quark shopping queries. Video: web clips kept only when a vision-language model,
shown sampled frames, agrees they match, with the query then corrected to what is visible.
Audio: event, instrument and command labels, captions and transcripts, in both directions.
Text: five retrieval paradigms, domain data in law, finance, code and health, and MTEB-style
tasks recast as retrieval with BGE-M3 hard negatives. Agents: tool, GUI and evidence retrieval,
with [BM25](/articles/bm25) negatives. The total is "approximately 50M" pairs, under a footnote
calling that paragraph's numbers "placeholder estimates based on the current data freeze".
**What the paper does not say.** Which teacher models were distilled from; $\tau$, $\gamma$,
$\lambda_{\min}$, $\lambda_{\max}$, the LoRA rank, batch sizes or step counts; or the compute,
beyond "clusters equipped with NVIDIA H100 80 GB GPUs". Its evaluation section opens by promising
to show "which components of the proposed recipe contribute the most", and then there is no
ablation of the focal loss, homogeneous sampling, distillation or LoRA-first training. Of seven
tables, the only component with numbers behind it is the elastic projection. Each stage is
plausible; that each earns its place is asserted, not shown.
## Shorter vectors
Matryoshka training ([arXiv:2205.13147](https://arxiv.org/abs/2205.13147)) makes prefixes of
one embedding usable as shorter embeddings, but the paper reports that adding it to the
multi-objective training hurt the full-width vector. So the shortening is done after training,
on a frozen encoder. First a rotation: the second moments of each modality's candidate
embeddings are estimated on equal-sized samples, averaged with uniform weights, and
eigendecomposed once, $\Sigma = V \Lambda V^{\top}$. Rotating by the orthogonal $V$ changes no
inner product; it only puts the high-variance directions first, where truncation keeps them. Then
a residual linear adapter per width, initialised at zero and fitted without labels to keep
each candidate's similarities to the whole batch and to its nearest neighbours as they were at
full width. Both fold into one matrix per width:
$$
\mathbf{z}_d(\mathbf{v}) = \frac{P_d\,\mathbf{v}}{\lVert P_d\,\mathbf{v} \rVert_2},
\qquad
P_d = \big[(I + W_d)\,V^{\top}\big]_{1:d,\,:}
$$
The paper's Table 7 (reported) holds the result. Averaged over the six suites with MMEB-v3's task
counts, the adapted embedding keeps **100.1%** of its full-width score at 1,024 dimensions,
**99.2%** at 512, **97.4%** at 256 and **93.2%** at 128. Cutting the same vector to its first $d$
coordinates and renormalising keeps 99.3%, 97.0%, 93.8% and 85.8%. The cost at 128 is uneven:
audio drops 0.33 points and video 1.26, while visual documents drop 6.97 and agent retrieval
6.18. The paper's reading is that finding one passage on a busy page, or one changed button in a
GUI, lives in low-variance directions that a short prefix throws away.
Two caveats. The 2,048 column of Table 7 averages **58.00**, not the **58.46** of the headline
table. The likely reason is that the adapter was fitted on the stage-2 encoder (Figure 5 labels
it "frozen Stage-2 encoder"), while the model card describes the released weights as the product
of all three training stages, so an adapter for them would have to be refitted (reasoned). And no adapter or projection matrix is in any of the three repos
(measured, from the file listings). What you can do with the released checkpoint today is
naive truncation, the 85.8%-at-128 curve.
## What MMEB-v3 measures
MMEB-v3 ([arXiv:2604.23321](https://arxiv.org/abs/2604.23321)) comes from authors at the
University of Waterloo, Shanghai Jiao Tong, the Eastern Institute of Technology in Ningbo and
Google, among others, and extends MMEB-v2's image, video and visual-document suites with 111 new
tasks. The leaderboard is a Hugging Face Space run by TIGER-Lab, fed by score files that teams
upload; a file carries a `data_source` field, and one without it is shown as self-reported. The
counts, from the leaderboard's `datasets.py` (measured):
| Group | Tasks | Metric | Share of overall |
|---|---:|---|---:|
| Image (classification, VQA, retrieval, grounding) | 37 | Hit@1 | 19.5% |
| Video (classification, QA, retrieval, moment retrieval) | 18 | Hit@1 | 9.5% |
| Visual documents (ViDoRe v1/v2, VisRAG, out-of-domain) | 24 | nDCG@5 | 12.6% |
| Text (FollowIR, R2MED, InfoSearch, BRIGHT, LongEmbed, MultiConIR, NanoBEIR) | 53 | nDCG@5 | 27.9% |
| Audio (5 classification, 6 retrieval) | 11 | Hit@1 | 5.8% |
| Agent (tool, GUI, memory retrieval) | 47 | Hit@1 | 24.7% |
The overall score is the unweighted mean over all 190 tasks, with a task a model cannot run
counted as zero. So text and agent retrieval are 100 of the 190 tasks and audio is 11. An
"omni-modal" number is mostly a text-and-agent number with images on the side, and the audio
group, the reason to use an omni backbone at all, moves the total by at most 5.8% of its range.
## The numbers, checked
I reimplemented the leaderboard's aggregation (its `utils_v3.py`, read, not run) over its score
files at commit `8129607` of 16 September. Ovis's own file, uploaded 3 September, recomputes to
**58.46** overall, and every group score in the paper's Table 1 is within 0.01 of the files
(measured). Three of the four baselines are files marked "Reproduced by TIGER-Lab": e5-omni-7B,
Omni-Embed-Nemotron-3B and LCO-Embedding-Omni-7B. The fourth, Tianmu-Emb-Uni, and Ovis itself
are self-reported. The baselines are the leaderboard's numbers, not picks from each model's own
paper, and the headline against them holds: **58.46** against Tianmu's **53.27**, e5-omni-7B's
**47.14**, Nemotron's **43.60** and LCO's **43.14**, first in all six groups (reported, and
recomputed).
The files also hold models the paper did not compare against, and two of them matter (measured):
- **WeMM-Embedding-9B** from Tencent, a Qwen3.5-based vision-language model uploaded on 26 August,
recomputes to **59.33** overall (59.55 before a correction to two of its agent tasks on 16
September). It cannot take audio, so its 11 audio tasks count as zero, and it still finishes
ahead. Over the 179 non-audio tasks, it is **62.98** to Ovis's **58.97**. It is the only file
that beats Ovis's **58.46**, and it is three times Ovis's size.
- **AuroLA-Omni-7B**, uploaded the same day, scores **50.49** on audio to Ovis's **50.08**: lower
on audio classification (71.07 to 73.30), higher on audio retrieval (33.34 to 30.73).
Both files predate the paper by nearly four weeks. "Ranks first on the aggregate score of every
evaluation group" is true of the four baselines chosen, and not of the leaderboard.
**Where it is weak.** Audio retrieval averages **30.73** Hit@1 against **73.30** for audio
classification; per task (measured) that is 21.53 on Clotho, 14.43 on AVE and 5.61 on
TUTSound's hard temporal-grounding split. Putting a label on a clip works; finding the right
clip from a description mostly does not. Video retrieval sits at **52.05**. On text,
reasoning-heavy BRIGHT is **15.66** and medical R2MED **25.68**, and MultiConIR, queries with
several conditions that must all hold, is **61.73**, 7.94 behind Nemotron. One cosine has to
fold every condition into one number; [the jev-semgrep piece](/articles/search-by-meaning) makes
the related point that similarity cannot say "and not". Agent memory retrieval, **29.44**, is
second to Nemotron's 32.23.
**Two slips in the paper**, besides the placeholder footnote, neither of which changes the
ranking (measured against its own tables):
- The text puts the audio margin at 7.04 points. That is the gap to e5-omni-7B's 43.04; the
runner-up in Table 1 is LCO-Embedding-Omni-7B at 43.17, so the margin is **6.91**, which is
what the model card says.
- Figure 1's MMEB-v3 panel, below, replaces Tianmu, the strongest baseline in Table 1, with
Qwen3-VL-Embedding-2B, whose bars read 75.0 on image and 61.9 on video. Those are its
**MMEB-v2** scores from the paper's own Table 6 (74.96, 61.87). The MMEB-v3 paper reports 69.5
and 55.9 for it.
## Beyond MMEB-v3
**Audio and video suites** (reported). On the beta Massive Audio Embedding Benchmark, 30 tasks,
Omni-3B averages **57.29** per task against LCO-Embedding-Omni-7B's 53.54. On the beta Massive
Video Embedding Benchmark, 23 tasks, it averages **61.77** against LCO-7B's 57.58. Both are the
team's local runs, not leaderboard submissions; the ranks in the paper's tables were estimated by
inserting them into a leaderboard snapshot. The weak spots rhyme with MMEB-v3's: audio
clustering at 17.59 (the best in the table, and still low in absolute terms), audio reranking
80.70 against e5-omni-7B's 86.70, video clustering 25.35 against LCO-7B's 27.35.
**Text** (reported). There is no MTEB score. On RTEB's 15-task English public split, Omni-3B
scores **67.35** against Qwen3-Embedding-4B's **67.27**, which is a tie, not a win. Underneath,
it is 49.61 to 62.67 on legal and 80.00 to 69.75 on finance, and the paper says it trained on
data built for exactly RTEB's domains, law, finance, programming and healthcare, while
deduplicating against the test sets. The "MMEB-Text 47.15" in the same table is MMEB-v3's own
53-task text group again, not a separate benchmark.
**The VL siblings on MMEB-v2** (78 tasks, no audio). VL-9B scores **81.13**, first among every
file on the leaderboard I recomputed (measured). Its margin over the strongest model it was
compared with, Octen-VL-Embedding-Large, is 1.04; over WeMM-Embedding-9B, which it was not
compared with, 0.55. On video it trails Octen-Large, 72.90 to 75.95. VL-2B scores **77.46**,
and WeMM-Embedding-2B, not in its comparison, has **77.94** in the files.
## Practicalities
**Licence.** All three repos say Apache-2.0. Qwen3.5-2B and 9B are Apache-2.0, so the VL chain is
clean. Qwen2.5-Omni-3B is not: its repo carries the Qwen Research License, which grants use
"FOR NON-COMMERCIAL PURPOSES ONLY" and lets derivatives carry their own terms only where use
"otherwise complies" with it. Alibaba licensed the base and Alibaba released the derivative, so
it may be entitled to relicense, but a README is not where that gets settled. For a product, get
the Omni licence confirmed in writing, or use a VL model and give up audio (reasoned; not legal
advice).
**Code.** None. The GitHub repos hold a README and a logo, and both READMEs still say the
weights "are not open-sourced yet", though they are on Hugging Face. The model cards describe the
retrieval steps in prose, with no snippet and no instruction strings beyond the Figure 2 example.
An `args.json` in each checkpoint names a template, `qwen2_5_omni_emb` or `qwen3_5_emb`, with
`task_type: embedding`, which reads like MS-SWIFT's format; the paper mentions MS-SWIFT when it
discusses training speed (reasoned).
**What an input costs**, reasoned from the processor configs and the Qwen2.5-Omni report:
| Input | Omni-3B | VL-2B / VL-9B |
|---|---|---|
| Image | one token per 28 × 28 px (patch 14, 2 × 2 merge); 1024 × 1024 → 1,369 tokens, 1920 × 1080 → 2,691 | one token per 32 × 32 px (patch 16, 2 × 2 merge); 1024 × 1024 → 1,024, 1920 × 1080 → 2,040 |
| Image bounds | 4 to 16,384 tokens | 64 to 16,384 tokens |
| Audio | 16 kHz, 128-bin mel, about 40 ms per token: 25 tokens per second, 750 for a 30 s clip | not supported |
| Video | two frames share one token grid: at 2 fps and 448 × 448, 256 tokens per second, plus 25 per second of soundtrack | same pairing at 32 px per token |
A 1024-pixel square image costs Omni-3B roughly 7.6 TFLOP in the Thinker (twice its 2.77B
non-embedding parameters, times 1,369 tokens) plus about 7.3 in the vision encoder, which runs
on the 5,476 patches before merging: two FLOPs per parameter per token, attention ignored
(reasoned). A 30-token text query is about 0.17 TFLOP, so indexing images costs roughly ninety
times what indexing short captions does.
**Storage.** A 2,048-dimensional fp16 vector is 4,096 bytes, so a million of them are 4.1 GB
before any ANN structure; VL-9B's 4,096 dimensions double that. Naive truncation to 512
dimensions, keeping 97.0% of the score on Table 7's stage-2 numbers, is a quarter of the size.
## What I would use it for
For an any-to-any index where audio has to sit next to everything else, this is the strongest
open model in the leaderboard's files that takes audio at all, and at 3B the smallest of the top
three. The mechanism is cheap to reason about: one forward pass, the last token, a cosine. Be
careful where the paper is quiet: audio retrieval, unlike audio classification, is still weak;
Table 1 is honest about its four baselines and silent about a stronger model that was already
public; the recipe has no ablations; the checkpoint carries 1.14B parameters it never uses; and
the licence chain runs through a research-only base. If you do not need audio, VL-9B has the
cleaner licence and the top MMEB-v2 score, and WeMM-Embedding-9B deserves a run on your own data
before you choose.