2026-07-17 · 8 min · multimodal · video-understanding · vision-language · efficiency · explainer
VideoChat3, from Nanjing University, Shanghai AI Lab, NTU, and Peking University (arXiv 2607.14935, HF), is a 4B-parameter video multimodal LLM with one clear thesis: you should not have to pick between a model that generalizes across video types, a model that is cheap to run on long clips, and a model you can actually reproduce. Most video MLLMs give you one of the three. This one aims for all three, and ships the training data to prove the last.
The interesting part is not the leaderboard — it is how a 4B model stays coherent over 2048 frames without the token count exploding. Two mechanisms carry that: an inflated 3D tokenizer, and adaptive frame resolution. I'll walk both, then the numbers.
The problem: tokens grow with time
A video is frames, and every frame is a few hundred vision tokens. Feed a clip in frame-by-frame and the sequence length grows linearly with duration — a few thousand tokens for a short clip, tens of thousands for a long one. That is what makes long video expensive: the LLM pays quadratic attention over a token budget set by the tokenizer, and the tokenizer, if it treats each frame independently, has no reason to be frugal.
Two failure modes fall out of that. Compute blows up on long clips. And a 2D image tokenizer, bolted on frame-by-frame, never models motion — it sees a stack of stills. VideoChat3 attacks both at the tokenizer.
I3D-ViT: inflating a 2D tokenizer into 3D
The first move is the Inflated 3D Vision Transformer (I3D-ViT). Start from a 2D image encoder — a plain patch-and-attend ViT — and inflate it into a spatiotemporal one instead of training a video encoder from scratch. The inflation is three steps:
- Chunk the frames. Group
T = 4consecutive frames into a chunk. - Attend within the chunk. Run self-attention across the chunk's tokens, so spatial and temporal structure are modelled together — motion, locally.
- Pool the chunk down. A temporal pooling (×4) plus a 2×2 spatial merge collapse each chunk to a compact, motion-aware token slot.
Temporal ÷4 times spatial ÷4 is a 16× spatiotemporal compression — the token budget grows with the frame count, but 16× slower than the naive per-frame path. Drag the frame count and watch where the tokens go:
The tokenizer never sees frames one at a time. It groups 4 consecutive frames into a chunk, runs self-attention inside the chunk so motion is modelled locally, then pools the chunk down to a single temporal slot — 64 spatial tokens per frame become 4 per frame after the 2×2 merge and ×4 temporal pool. A 16× spatiotemporal compression, and the token budget grows with the frame count instead of exploding with it.
The reason this works without wrecking accuracy: the compression happens after the chunk has been attended, so motion is already encoded into the surviving tokens. You are not throwing frames away — you are summarising each 4-frame window into one slot that remembers what moved. Native resolution and aspect ratio are preserved through absolute temporal embeddings, so the model still knows when each token happened.
Adaptive frame resolution: spend pixels where the evidence is
Compression handles the count of tokens. The second move handles the cost per frame. In a streaming setting most of a video is uneventful — a static room, a held shot, dead air. Processing every frame at high resolution spends the same budget on the boring frames as on the one that answers the question.
So VideoChat3 conditions the per-frame resolution on state. Routine moments are perceived under a low 224²-pixel quota. When a Standby cue fires — the signal that an answer is about to appear — the following window is enlarged to a 448² quota, roughly 4× the tokens, to catch the detail. Click frames to promote them and watch the budget move:
Most of a stream is silence — nothing worth spending pixels on — so those frames run at a 224² quota. A Standby cue says the answer is about to appear; the frames that follow jump to 448² (4× the tokens) to catch the detail. Spend the budget where the evidence is, not uniformly. The failure mode: mark too few frames critical and a fast event is seen only in low-res.
The framing is a three-state stream: Silence (nothing to report, low-res), Standby (something is coming, stay ready), Response (answer now, high-res). The budget follows the state instead of the clock. On a stream that is mostly silence, that is most of the frames spent at a quarter of the token cost.
The benchmarks
The headline: at 4B parameters, VideoChat3 beats comparable open models (Qwen3-VL-4B, Molmo2-4B) across temporal perception, long video, reasoning, temporal grounding, and online tasks — the paper's cross-benchmark sweep is one figure:
Where it separates most is temporal grounding — answering when something happens, not just what. Over Qwen3-VL-4B the gains run from +9.7 (Charades) up to +20.6 (VUE-TR V2 in the TimeLens suite), depending on the benchmark. Charades makes the point:
The same ordering holds on ActivityNet (54.8 / 48.2 / 39.8) and QVHighlights (67.1 / 58.7 / 58.7). It is not just grounding, though — the general video and reasoning benchmarks land ahead too, if by smaller margins:
VideoMME 70.1 (vs 69.3 / 69.6), LVBench 56.7 (vs 56.2 / 53.9), TempCompass 75.6 (vs 70.8 / 72.8), StreamingBench 83.0 (vs 80.2). The deltas on the general suites are single digits; the grounding deltas are where the tokenizer design shows up.
The efficiency payoff
The point of 16× compression is latency, and it compounds with length. Same hardware, same clip, VideoChat3 vs Qwen3-VL, end-to-end inference:
| Frames | Qwen3-VL | VideoChat3 |
|---|---|---|
| 512 | 3.84s | 3.60s |
| 1024 | 12.25s | 8.10s |
| 2048 | 44.45s | 20.41s |
At 512 frames the gap is small — the tokenizer overhead is a rounding error. At 2048
frames it is 2.2×: 44.4s → 20.4s. The compression buys you the frames the
grounding benchmarks reward, at a latency that stays usable as the clip grows.
Fully open: the data, not just the weights
The "fully open" claim is the part I'd flag to anyone who has tried to reproduce a video MLLM. The bottleneck is never the architecture — it is the ~3M-sample instruction mix nobody publishes. VideoChat3 releases three:
- VideoChat3-Academic2M — 2.27M caption/QA instances from six academic sources, with evidence-grounded annotation enhancement.
- VideoChat3-LV116K — 116.2K long-form samples, mean durations 156s to ~1.3K seconds.
- VideoChat3-OL617K — 617,183 streaming/online instances across 40 shards.
Trained through a four-stage curriculum: tokenizer pretraining → video-language alignment → general instruction tuning → long/streaming tuning. Weights and data both out, so the recipe is checkable end to end.
What I make of it
- The tokenizer is the whole story. I3D-ViT is a clean idea — inflate a 2D encoder, attend within short chunks, pool 16×. It is why a 4B model can watch 2048 frames in 20 seconds, and why the temporal-grounding gaps are as large as they are. Motion survives the compression; that is the trick.
- Adaptive resolution is the right shape for streaming. Spend the budget on the evidence, not the clock. It maps cleanly onto a Silence/Standby/Response state machine, and the savings are largest exactly where video is cheapest to skimp — the dead air.
- Open data is the contribution that outlasts the benchmarks. Numbers age; a released 3M-sample video instruction mix is something the rest of the field can build on. For a model whose pitch is "generalist and reproducible," shipping Academic2M + LV116K + OL617K is the part that makes the claim real.
Source: "VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding," Li, Zhu, Zeng, Dong, Wu, et al. (arXiv 2607.14935). Benchmark values read from the paper's reported figures and tables; numbers quoted as reported. Interactive diagrams are my own illustration of the mechanism — token counts are illustrative, ratios are the paper's.