# VideoChat3: a 4B video model that watches longer for less

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/videochat3
> date: 2026-07-17
> tags: multimodal, video-understanding, vision-language, efficiency, explainer
VideoChat3, from Nanjing University, Shanghai AI Lab, NTU, and Peking University
([arXiv 2607.14935](https://arxiv.org/abs/2607.14935),
[HF](https://huggingface.co/papers/2607.14935)), is a **4B-parameter video
multimodal LLM** with one clear thesis: you should not have to pick between a model
that generalizes across video types, a model that is cheap to run on long clips, and
a model you can actually reproduce. Most video MLLMs give you one of the three. This
one aims for all three, and ships the training data to prove the last.

The interesting part is not the leaderboard — it is *how* a 4B model stays coherent
over 2048 frames without the token count exploding. Two mechanisms carry that: an
inflated 3D tokenizer, and adaptive frame resolution. I'll walk both, then the
numbers.

## The problem: tokens grow with time

A video is frames, and every frame is a few hundred vision tokens. Feed a clip in
frame-by-frame and the sequence length grows linearly with duration — a few thousand
tokens for a short clip, tens of thousands for a long one. That is what makes long
video expensive: the LLM pays quadratic attention over a token budget set by the
tokenizer, and the tokenizer, if it treats each frame independently, has no reason to
be frugal.

Two failure modes fall out of that. Compute blows up on long clips. And a 2D image
tokenizer, bolted on frame-by-frame, never models *motion* — it sees a stack of
stills. VideoChat3 attacks both at the tokenizer.

## I3D-ViT: inflating a 2D tokenizer into 3D

The first move is the **Inflated 3D Vision Transformer (I3D-ViT)**. Start from a 2D
image encoder — a plain patch-and-attend ViT — and *inflate* it into a spatiotemporal
one instead of training a video encoder from scratch. The inflation is three steps:

1. **Chunk the frames.** Group `T = 4` consecutive frames into a chunk.
2. **Attend within the chunk.** Run self-attention across the chunk's tokens, so
   spatial and temporal structure are modelled together — motion, locally.
3. **Pool the chunk down.** A temporal pooling (×4) plus a 2×2 spatial merge collapse
   each chunk to a compact, motion-aware token slot.

Temporal ÷4 times spatial ÷4 is a **16× spatiotemporal compression** — the token
budget grows with the frame count, but 16× slower than the naive per-frame path.
Drag the frame count and watch where the tokens go:

<I3dVit />

The reason this works without wrecking accuracy: the compression happens *after* the
chunk has been attended, so motion is already encoded into the surviving tokens. You
are not throwing frames away — you are summarising each 4-frame window into one slot
that remembers what moved. Native resolution and aspect ratio are preserved through
absolute temporal embeddings, so the model still knows *when* each token happened.

<Figure
  src="/articles/videochat3/fig1.png"
  alt="I3D-ViT pipeline: native-resolution frames are patchified, given spatial and temporal positional embeddings, passed through variable-length self-attention within 4-frame chunks, then chunk-wise temporal pooling and a 2x2 pixel-shuffle merge feed compact video tokens into the LLM."
  caption="I3D-ViT inflates a 2D tokenizer: patchify → spatial + temporal position embeddings → variable-length self-attention inside frame chunks → temporal pooling + 2×2 merge → compact tokens into the LLM (Li et al., 2026, Figure 2)."
/>

## Adaptive frame resolution: spend pixels where the evidence is

Compression handles the *count* of tokens. The second move handles the *cost per
frame*. In a streaming setting most of a video is uneventful — a static room, a
held shot, dead air. Processing every frame at high resolution spends the same budget
on the boring frames as on the one that answers the question.

So VideoChat3 conditions the per-frame resolution on state. Routine moments are
perceived under a low **224²-pixel** quota. When a *Standby* cue fires — the signal
that an answer is about to appear — the following window is enlarged to a **448²**
quota, roughly 4× the tokens, to catch the detail. Click frames to promote them and
watch the budget move:

<AdaptiveResolution />

The framing is a three-state stream: **Silence** (nothing to report, low-res),
**Standby** (something is coming, stay ready), **Response** (answer now, high-res).
The budget follows the state instead of the clock. On a stream that is mostly
silence, that is most of the frames spent at a quarter of the token cost.

<Figure
  src="/articles/videochat3/fig3.png"
  alt="Streaming timeline: clips 0 through N+2 are processed low-res during silence, clips N+3 and N+4 jump to high-res as a Response window opens, then clip N+5 drops back to low-res."
  caption="Adaptive perception on a live stream: low-res while Silence holds, high-res for the Response window, back to low-res after — the token quota tracks the state, not the frame index (Li et al., 2026, Figure 3)."
/>

## The benchmarks

The headline: at **4B parameters**, VideoChat3 beats comparable open models
(Qwen3-VL-4B, Molmo2-4B) across temporal perception, long video, reasoning, temporal
grounding, and online tasks — the paper's cross-benchmark sweep is one figure:

<Figure
  src="/articles/videochat3/fig2.png"
  alt="Grouped bar chart comparing Molmo2-4B, Qwen3-VL-4B, and VideoChat3 across MotionBench, TempCompass, VideoMME, LVBench, MMVU, VideoMME-v2, Charades, ActivityNet, QVHighlights, OVOBench, and StreamingBench; VideoChat3 leads on all, with large margins on the temporal-grounding benchmarks."
  caption="VideoChat3 vs Qwen3-VL-4B and Molmo2-4B across eleven benchmarks — leading everywhere, with the widest gaps on temporal grounding (Li et al., 2026, Figure 1)."
/>

Where it separates most is **temporal grounding** — answering *when* something happens,
not just *what*. Over Qwen3-VL-4B the gains run from **+9.7** (Charades) up to **+20.6**
(VUE-TR V2 in the TimeLens suite), depending on the benchmark. Charades makes the point:

<BenchBars
  title="Temporal grounding — Charades (mIoU)"
  unit=""
  bars={[
    { label: "VideoChat3-4B", value: 56.1, highlight: true },
    { label: "Qwen3-VL-4B", value: 46.4 },
    { label: "Molmo2-4B", value: 33.3 },
  ]}
/>

The same ordering holds on ActivityNet (54.8 / 48.2 / 39.8) and QVHighlights
(67.1 / 58.7 / 58.7). It is not just grounding, though — the general video and
reasoning benchmarks land ahead too, if by smaller margins:

<BenchBars
  title="Video reasoning — MMVU (%)"
  unit=""
  bars={[
    { label: "VideoChat3-4B", value: 56.4, highlight: true },
    { label: "Molmo2-4B", value: 51.2 },
    { label: "Qwen3-VL-4B", value: 50.5 },
  ]}
/>

VideoMME 70.1 (vs 69.3 / 69.6), LVBench 56.7 (vs 56.2 / 53.9), TempCompass 75.6 (vs
70.8 / 72.8), StreamingBench 83.0 (vs 80.2). The deltas on the general suites are
single digits; the grounding deltas are where the tokenizer design shows up.

## The efficiency payoff

The point of 16× compression is latency, and it compounds with length. Same
hardware, same clip, VideoChat3 vs Qwen3-VL, end-to-end inference:

| Frames | Qwen3-VL | VideoChat3 |
|---|---|---|
| 512 | 3.84s | 3.60s |
| 1024 | 12.25s | 8.10s |
| 2048 | 44.45s | **20.41s** |

At 512 frames the gap is small — the tokenizer overhead is a rounding error. At 2048
frames it is **2.2×**: `44.4s → 20.4s`. The compression buys you the frames the
grounding benchmarks reward, at a latency that stays usable as the clip grows.

## Fully open: the data, not just the weights

The "fully open" claim is the part I'd flag to anyone who has tried to reproduce a
video MLLM. The bottleneck is never the architecture — it is the ~3M-sample
instruction mix nobody publishes. VideoChat3 releases three:

- **VideoChat3-Academic2M** — 2.27M caption/QA instances from six academic sources,
  with evidence-grounded annotation enhancement.
- **VideoChat3-LV116K** — 116.2K long-form samples, mean durations 156s to ~1.3K
  seconds.
- **VideoChat3-OL617K** — 617,183 streaming/online instances across 40 shards.

Trained through a four-stage curriculum: tokenizer pretraining → video-language
alignment → general instruction tuning → long/streaming tuning. Weights and data
both out, so the recipe is checkable end to end.

<Callout type="warn">
Read the comparison for what it is: a **4B-vs-4B** result. VideoChat3 beats *comparable
open* models at its size — it is not claiming to beat frontier closed video systems or
much larger open ones, and the general-suite margins (VideoMME +0.5 to +0.8) are
inside the range where mix and eval harness matter. The token math here is
illustrative (I use ~64 spatial tokens/frame to keep the diagrams honest about
*ratios*, not absolute counts); the 16× compression, the 224²/448² quotas, and the
latency numbers are the paper's. Adaptive resolution has a real failure mode too: set
the Standby threshold too tight and a fast event is only ever seen in 224².
</Callout>

## What I make of it

- **The tokenizer is the whole story.** I3D-ViT is a clean idea — inflate a 2D
  encoder, attend within short chunks, pool 16×. It is *why* a 4B model can watch 2048
  frames in 20 seconds, and *why* the temporal-grounding gaps are as large as they are.
  Motion survives the compression; that is the trick.
- **Adaptive resolution is the right shape for streaming.** Spend the budget on the
  evidence, not the clock. It maps cleanly onto a Silence/Standby/Response state
  machine, and the savings are largest exactly where video is cheapest to skimp — the
  dead air.
- **Open data is the contribution that outlasts the benchmarks.** Numbers age; a
  released 3M-sample video instruction mix is something the rest of the field can build
  on. For a model whose pitch is "generalist *and* reproducible," shipping
  Academic2M + LV116K + OL617K is the part that makes the claim real.

---

*Source: "VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video
Understanding," Li, Zhu, Zeng, Dong, Wu, et al.
([arXiv 2607.14935](https://arxiv.org/abs/2607.14935)). Benchmark values read from the
paper's reported figures and tables; numbers quoted as reported. Interactive diagrams
are my own illustration of the mechanism — token counts are illustrative, ratios are
the paper's.*