LeVJEPA: one encoder, zero collapse-prevention machinery, and what 5.6-20.8x actually measures

Every joint-embedding video model has the same problem to solve before it solves anything else: an encoder trained to make two views of a clip agree will happily satisfy that objective by mapping everything to the same constant vector. V-JEPA 2 — the video world-model backbone Meta trained for planning and action-conditioned prediction — solves it the way BYOL and DINO do: an exponential-moving-average target encoder, a stop-gradient so the target never sees a training signal, and a predictor narrow enough that it can't just memorize the answer. Masked-video methods like VideoMAE sidestep the question entirely by reconstructing pixels, which admits no collapsed solution to begin with — at the cost of a decoder and masking schemes tuned around the imputation task rather than around video.

LeVJEPA (Kuhn, Maes, Serra, Le Lidec, LeCun, Balestriero, Buettner — DKFZ, Goethe, Mila, Université de Montréal, Brown, NYU Courant, and AMI Labs; arXiv 2608.27395) is the first video encoder trained under neither machinery. It transfers LeJEPA — Balestriero and LeCun's collapse-free objective for images, published three months before this paper — to video, and reports that the video-specific baggage (temporal masking heuristics, an asymmetric target branch, a predictor) turns out not to be load-bearing. What's left is one encoder, one loss, and — genuinely — one hyperparameter. This piece checks the paper's five headline numbers against its own tables and against the released ViT-L checkpoint's config, safetensors header, and commit history, and finds most of them hold up more precisely than the abstract states them.

The objective: an encoder that grades its own homework

From each 16-frame clip, LeVJEPA builds one global view at full resolution and $V$ local views — aggressive spatial crops with photometric augmentation, all sharing the clip's exact temporal window (default $V = 4$ for every comparison in the paper). Every view goes through the same encoder $E_\theta$; a learnable [cls] token gives a clip-level readout, which a small projector maps to an embedding $z_v \in \mathbb{R}^K$. The training loss is

$z_0$ is the global view's own embedding — the "target" is not a separate network's output, it's this same encoder's output on a different view, in the same forward pass. Minimized alone, that invariance term has a trivial fix: output the same constant vector for every input, and the loss goes to zero. LeJEPA's answer is SIGReg — a regularizer that constrains the batch of embeddings toward an isotropic Gaussian, the distribution LeJEPA shows minimizes worst-case downstream probing risk, and one that no collapsed (zero-variance-in-some-direction) solution can approximate. By the Cramér–Wold theorem, a high-dimensional Gaussian-matching constraint reduces to univariate goodness-of-fit tests along random projections, computed with the Epps–Pulley statistic — 1,024 random directions and a 17-point quadrature per training step, per the paper's Appendix A. $\lambda = 0.02$ is LeJEPA's published default and is not tuned anywhere in this paper.

expand

That single sentence — same encoder, same forward pass, no separate target — is the whole simplification. Everything downstream in this piece is a consequence of it.

Two things about that table are easy to undercount. First, the projector is not a renamed predictor: V-JEPA's predictor is queried at masked positions as part of the pretraining task itself, while LeVJEPA's projector (a 2-layer MLP, $d \to 2048 \to K{=}256$, with batch norm and GELU) is discarded entirely after pretraining — it never runs at inference, and it isn't in the Hugging Face checkpoint. Second, the released weights do carry one averaging mechanism — a Polyak average of the encoder, decay 0.9999, updated every 32 optimizer steps, confirmed against the checkpoint's own model card. The paper is explicit that this is not a collapse-prevention component: "it receives no forward passes during training and does not appear in the objective," unlike an EMA target encoder that produces training targets every step. It's a postprocessing step applied to an objective that was already collapse-free without it.

How much cheaper, really — and is the range honest?

The abstract's headline is "5.6 to 20.8x less total pretraining compute" against V-JEPA 2, and — unlike a lot of multipliers this site has checked — that range is exactly what the abstract states, not a best cell dressed up as the whole story. The three points come from an epoch-matched protocol: ViT-S, ViT-B, and ViT-L encoders, each pretrained for 240 epochs on an identical 20% subsample of K710, with V-JEPA 2 retrained on the same data using its own official implementation and hyperparameters so the comparison isn't confounded by different pretraining corpora.

expand

Reading the exact numbers the paper states in prose rather than eyeballing the chart: the two endpoints of the range sit at opposite ends of the model-size axis, and the direction is worth being precise about. ViT-S is the 20.8x end — the smallest encoder, where LeVJEPA and V-JEPA 2 land within noise of each other on accuracy. ViT-L is the 5.6x end, and it's the more interesting cell precisely because the ratio is smaller: at ViT-L, LeVJEPA doesn't just match V-JEPA 2 at lower cost, it beats it by 1.9 accuracy points — and that ViT-L still uses less compute than V-JEPA 2's own ViT-S. ViT-B sits in between at a ratio the paper states in absolute terms rather than a multiplier: 4.8 ExaFLOPs for LeVJEPA against 36.4 for V-JEPA 2 (a 7.6x ratio computed from those two numbers), with the methods separated by less than one accuracy point. So the honest reading of "5.6 to 20.8x" is that it's a real range across three model sizes, not one flattering cell — but it also means the two ends of that range are winning in different ways: ViT-S wins on compute ratio alone, ViT-L wins on both compute and accuracy simultaneously.

VideoMAEv2, run under the same protocol, lands in between both methods on compute cost and below both on accuracy at this epoch-matched setting — worth naming because it reappears as the stronger baseline in the next comparison, under a different protocol entirely.

Two different questions get two different tables

It's easy to fold "V-JEPA 2 at 5.6-20.8x less compute" and "+7.6 points on ImageNet-1K" into one finding. They're not the same experiment. The compute-multiplier numbers above hold the number of training epochs fixed and let total FLOPs vary by method. The +7.6-point number instead holds total FLOPs fixed and lets epochs vary — and because LeVJEPA processes far fewer tokens per sample, an equal FLOP budget buys it a proportionally longer schedule: 1,085 epochs at $V{=}10$ local views, versus 240 for the epoch-matched baselines above.

Under that FLOP-matched protocol, VideoMAEv2 — not V-JEPA 2 — is "the strongest video baseline" the abstract's +7.6 points is measured against: 61.0 vs. 53.4 on ImageNet-1K. V-JEPA 2 actually trails VideoMAEv2 here (51.6), so LeVJEPA's margin over V-JEPA 2 specifically is larger, 9.4 points — the paper reports the more conservative of the two gaps as its headline. "Remaining competitive on motion-centric benchmarks" is the Something-Something-v2 row: LeVJEPA is not the leader there, trailing VideoMAEv2 by 3.2 points (40.4 vs. 43.6). That's a real, bounded gap, not a rounding difference — the honest summary of the FLOP-matched table is a clean win on two benchmarks (IN1K, K400) and a moderate loss on the third (SSv2), not a sweep.

The DINOv2 comparison runs under the identical FLOP-matched logic, against an image encoder instead of a video one: DINOv2 trained with its official implementation on individual frames of the same video data, 11.7M frame samples over 11,400 steps, at the same total FLOPs as the 240-epoch LeVJEPA ViT-B. "Approaches the image-pretrained encoder on appearance-centric evaluation" is a 3.1-point gap on ImageNet-1K (53.8 vs. 50.7) — LeVJEPA reaches 94% of DINOv2's accuracy at equal compute, not full parity, but close enough that the paper's framing holds. "Nearly doubling its motion-centric accuracy" is 30.4% against 16.9%, a 1.80x ratio — genuinely close to double, and the more striking number of the two given DINOv2 never sees a moving frame relative to any other frame during training.

Block-causal attention: shipped, not just ablated

Because no branch asymmetry constrains the encoder's attention pattern, LeVJEPA can adopt a block-causal mask: patch tokens attend bidirectionally within their own frame and causally to preceding frames only, so a frame's representation never depends on anything that happens later in the clip. The released checkpoint's own modeling_levjepa.py makes one detail precise that neither the paper's figure nor the project page's demo spells out: "bidirectional" and "block-causal" aren't two strengths of the same mask, they're a mask and the absence of one. LeVJEPAModel.forward sets attn_mask = None unless attn_mode == "block_causal" — under bidirectional attention there is no mask object anywhere in the graph, every token attends to every other token symmetrically, [cls] included. Only the block-causal path introduces the asymmetric rule, and it introduces two rules at once: causality across frames for patches, and a [cls] token that behaves as a read-only sink — attending to the whole clip while no patch attends back to it, a design choice the mask function's own docstring explains directly (preventing exactly the kind of future-to-past information leak causal masking exists to stop).

The accuracy comparison (Table 2 of the paper, frozen attentive probe, $\tau{=}1$, $\rho{=}0.95$, $V{=}4$) is bidirectional 50.7% against block-causal 51.2% — so "no measurable accuracy cost" actually undersells it slightly; block-causal comes out 0.5 points ahead, within what's plausibly noise but never behind. And this isn't an ablation left in the paper for completeness: the released Hugging Face checkpoint's config.json sets "attn_mode": "block_causal" as the shipped default, with the model card warning that running the weights under full attention "will not raise an error — it will quietly return worse features." What ships is the causal encoder, not the more conventional bidirectional one.

That causality also isn't just an accuracy-neutral curiosity — it's what the paper's discussion section argues is the more consequential result. A frame representation computed only from past frames means a video's representation can extend incrementally as new frames arrive, without re-encoding anything that came before: a property autoregressive world models and streaming inference need, and one that bidirectional encoders can currently only approximate by re-encoding the whole clip or fitting a separate temporal model after the fact (as V-JEPA 2-AC does, training a causal predictor on top of a frozen bidirectional V-JEPA 2 encoder). LeVJEPA's version of that property is built into the encoder's own attention pattern during pretraining, not bolted on afterward.

Token dropping is a training-time lever, not an inference one

The paper's most counterintuitive result is that dropping tokens improves accuracy rather than merely making training cheaper. A fraction $\rho$ of patch tokens is discarded uniformly at random after patch embedding, and only the survivors enter the encoder. If this were purely an efficiency approximation, accuracy should degrade as $\rho$ grows; instead, ImageNet-1K accuracy rises monotonically:

Going from 90% to 95% dropped — halving the tokens processed a second time — leaves accuracy unchanged within noise, so the most aggressive setting tested is simultaneously the cheapest. The paper reads this as token dropping doing two jobs at once: cutting feed-forward cost by up to a factor of $(1-\rho)^{-1}$, and acting as a stochastic augmentation that forces the clip-level embedding to be inferable from a sparse, randomly placed sample of the clip. That second effect is also why the spatial pattern of what's kept matters more than how much is kept: a structured "tube" mask that retains identical spatial locations across all frames — the standard trick in masked-video reconstruction, where it exists to stop content being copied in from adjacent frames — actually hurts here, 39.6% against uniform random dropping's 50.7% on ImageNet-1K. With nothing being imputed, uniform random dropping leaves a spatio-temporally distributed sample the clip's content can still be recovered from; a tube permanently blacks out most of the scene in every frame. The reversal of a finding from a different objective, in a setting where that objective's justification no longer applies, is exactly the kind of result worth taking at face value rather than assuming it must generalize back.

None of this shows up at inference. The released checkpoint's config carries token_drop_rate: 0.0 and token_drop_mode: "random" — the fields exist so a training configuration round-trips through the same class, but they're inert once .eval() is called, and the model card says so plainly: "Token dropping is a training-time regulariser and is inert under eval(), so the released model returns all 3,137 tokens" (3,136 patches plus [cls]). A user loading this checkpoint for feature extraction gets the full, undropped sequence; the compute savings and the accuracy gain both belong entirely to pretraining, not to anything the checkpoint does when you call it.

Patch-level structure nobody supervised

The training objective reads only the [cls] embedding — patch tokens receive no direct loss, ever. The paper's next claim is that they organize themselves anyway:

expand

The comparison is specific and checkable: V-JEPA 2.1 gets this same kind of dense structure through "an explicitly introduced auxiliary patch-level objective" the paper names directly — it's not that patch-level structure is free everywhere, it's that other methods that have it paid for it with an extra loss term, and LeVJEPA gets a comparable result without one. V-JEPA 2, trained with neither a patch-level loss nor LeVJEPA's objective, is the control case in the same figure, and its decomposition is visibly unstructured — there's no free lunch being hidden in the comparison; the structure tracks the objective, not the architecture. The paper extends this with a cosine-similarity probe (a query patch placed on a moving object, checked against every other patch across frames): similarity stays confined to the object rather than diffusing across the frame, and — since the encoder is block-causal — that correspondence is computed from the current and preceding frames alone, not by attending forward into frames that haven't happened yet.

What actually shipped, one HF API call at a time

The paper reports results at ViT-S, ViT-B, and ViT-L. galilai-group has published exactly one of those three sizes on Hugging Face: LeVJEPA-VideoMix-Large — the ViT-L, matching the config's embed_dim: 1024, depth: 24, num_heads: 16, 303,099,904 parameters per the repo's own safetensors metadata. The license is cc-by-nc-4.0: non-commercial. Neither fact is a criticism — a lab publishing one checkpoint under a research license is a completely ordinary release shape — but both are worth stating plainly rather than assuming "the paper's numbers" and "what you can download" are the same offer.

Three smaller, more specific findings came out of reading the repository rather than the paper:

It ships in fp32. The safetensors header reports all 303.1M parameters as F32 — a ViT-L at full 32-bit precision, where most current open-weight releases ship bf16 or fp16 to halve the download and the memory footprint. config.json's own torch_dtype: "float32" confirms this is deliberate, not an upload artifact. Nothing in the model card explains the choice; it's simply not the default other labs have converged on for a checkpoint this size.

The advertised storage is exactly double the weights. Hugging Face reports usedStorage: 2,424,859,776 bytes (about 2.42 GB) for a repository whose only large file is one model.safetensors — but 303,099,904 F32 parameters is only about 1.21 GB, roughly half that figure. Reading the repo's commit history resolves this precisely rather than leaving it as a rounding mystery: the fourth commit, LeVJEPA-VideoMix-Large: ViT-L video encoder, EMA weights + modeling code, uploaded an initial model.safetensors; the fifth, titled plainly Replace weights: ep128 + 13-epoch 1-sqrt leg (in1k + SSv2-upweighted mixture), uploaded a second one under the same filename. Fetching both revisions' headers directly confirms they're genuinely distinct blobs (different x-linked-etag hashes) at the identical size, 1,212,429,888 bytes each — and $1{,}212{,}429{,}888 \times 2 = 2{,}424{,}859{,}776$, matching the reported storage figure exactly. The repository's usedStorage counts every unique blob ever pushed, not just the one reachable from main today; the original upload's weights are still sitting in storage, superseded but not deleted, because a "replace weights" commit is a new blob under git/LFS semantics, not an edit to the old one. It's a small piece of the release's own history that a plain API call surfaces without needing to download either file.

The commit message is itself informative. "ep128 + 13-epoch 1-sqrt leg (in1k + SSv2-upweighted mixture)" describes a checkpoint from epoch 128 of some run plus a further 13-epoch cosine-style ("1-sqrt") decay leg, trained on a mixture upweighted toward ImageNet-1K and Something-Something-v2 — consistent with, but not necessarily numerically identical to, the paper's headline "100 epochs on the combined K710 + SSv2 + Walking Tours + PE Video corpus" data-scaling result. The README's own training table (multi-crop objective, $V{=}10$ local views, 95% random token dropping, AdamW at a flat 4e-4 then a 1-sqrt decay to zero, batch 3,072, bf16-mixed precision) matches the paper's described recipe in every field it lists — but "ep128 plus a 13-epoch leg on an upweighted mixture" is a more specific description than "100 epochs on the union of four datasets," and nothing in the model card states the two are the same run. Worth knowing before assuming the checkpoint you download reproduces a specific number in the paper's tables to the decimal.

That data-scaling result is also where the paper and its own project page disagree with each other, independent of the checkpoint entirely. Both describe the identical experiment — a ViT-L/16 pretrained for 100 epochs on the combined K710 + Something-Something-v2 + Walking Tours + PE Video corpus, evaluated frozen — and both state the Something-Something-v2 result identically, 55.0%. But the arXiv HTML states the ImageNet-1K result as 69.5%, while the project page states 67.5% for the same run. This isn't a rounding difference or a units mismatch; it's a two-point gap between the paper's own archival text and its own promotional page, for a number both sources present as final. Only one arXiv revision exists (v1, no v2 to check for a correction), so there's no later version to resolve it against. This piece uses the arXiv figure since it's the citable record, but the discrepancy itself — not which number wins — is the checkable fact here.

Checked, in one table

The take

The mechanism here is genuinely simple to state and unusually well-verified for how simple it is: an encoder graded against its own other-view output, kept honest by a regularizer with a provable guarantee instead of an architectural trick, produces a video encoder that is both cheaper to train and — at the sizes and compute budgets tested — as good or better than one built the conventional way. The paper's own honesty helps here too: it states its compute-savings range up front rather than leading with the largest cell, names the one benchmark where it doesn't win, and reports a data-scaling run without smoothing over the fact that its own promotional page states a different number for it than the paper does.

What ships is narrower than what's reported — one size of three, non-commercial, fp32 for reasons the model card doesn't explain — but everything checkable about the shipped artifact (the block-causal default, the inert token-dropping config, even the storage byte-count) is consistent with the paper's own account of the method, right down to the exact EMA decay and update interval Appendix B specifies. For a self-supervised release, that level of internal consistency between the paper's claims and the artifact's own metadata is worth noting on its own — it's the kind of thing that's easy to get wrong by accident and this one doesn't.

For readers coming from the world-model side, LeVJEPA's block-causal, streaming-friendly encoder sits next to Cosmos 3's reasoner-generator pairing as a second, much smaller argument for building temporal causality into pretraining rather than fitting it on top of a frozen bidirectional backbone afterward. For readers coming from the video-understanding side, VideoChat3 tackles a related efficiency problem — compressing long video into fewer tokens — from the opposite end of the stack, an MLLM's tokenizer rather than a self-supervised encoder's objective. And for the attention-mask mechanics specifically, the site's field guide to attention variants is the place to see block-causal masking alongside the sliding-window, sink, and content-based alternatives it's one entry in.

Sources: LeVJEPA (arXiv 2608.27395v1), read via its arXiv HTML rendering; the project page; the LeVJEPA-VideoMix-Large model card, its config.json, configuration_levjepa.py, and modeling_levjepa.py, and its commit history and safetensors/LFS headers, all read directly via the Hugging Face API; LeJEPA (arXiv 2511.08544), Balestriero and LeCun; V-JEPA 2 (arXiv 2506.09985), Assran et al. Figures 1, 2, and 3 are the paper's own, fetched from its arXiv HTML rendering and shown for commentary. The collapse-machinery, block-causal mask, and FLOP-matched comparison diagrams are original, built from the sources above.