Where V-JEPA 2.1's Dense Features Hold Up (and Where They Don't)

May 6, 2026

Method

Nine perturbations at ten strengths on 200 SSv2 clips per cell across all four model sizes, one seed per cell, with six hypotheses and their decision rules pre-registered before launch; bootstrap 95% intervals on per-cell means, and within-perturbation Pearson r against tracking degradation on 30 DAVIS clips.

Result

Across four model sizes, dense features predict failure under temporal corruption and not under image noise, and the 2B model is less robust than the 1B on three of five perturbations.

Could not detect

One seed per cell, so between-clip variance is the only spread measured; intervals are bootstrap 95% and nothing narrower is claimed. Magnitudes across sizes, not paired comparisons: no verdict test and no minimum detectable effect apply.

Artifacts

code · cable-insertion code

We pre-registered a robustness study on Meta’s V-JEPA 2.1 (released March 2026) and ran it across all four released sizes, 80M to 2B parameters, a 322-cell sweep. Three things stand out. The dense features are partitioned: they predict downstream failure under temporal corruption (frame drops, occlusion, r = 0.35 to 0.37) and are indistinguishable from zero under image noise (Gaussian, motion blur, low light), so robustness on one axis doesn’t transfer to the other. Bigger isn’t reliably better: every Tier 1 perturbation was non-monotonic across the four sizes, and the 2B model is less robust than the 1B on three of the five. And the model is orientation-sensitive: a horizontal flip, which leaves all the temporal structure intact, disturbs the features about as much as playing the clip backwards.

Why we ran it

We’re integrating V-JEPA-family models as the perception backbone for two robotics workloads:

• Industrial Cable Insertion: Manipulation policies for sub-millimeter industrial cable insertion in cluttered environments. Visual conditions vary wildly across lighting, self-occlusion from the manipulator, and frame-rate variability under network constraints.
• Drone Infrastructure Inspection: Autonomous flight perception for tower and pipeline inspection, where camera roll is constant during maneuvers, motion blur is a perpetual factor, and low-light operations are common.

Why JEPA

V-JEPA 2.1 is positioned as a world model, a system with an internal representation of how the physical world behaves. Where a generative model spends its compute reconstructing pixels, JEPA predicts in a compressed latent space, which buys two things for robotics:

• Prediction about what’s happening rather than what it looks like, so the backbone can ignore lighting shifts and attend to the physics of the scene.
• Imagined futures: a robot can test what happens if it grabs an object from a given angle before it moves, and learn from imagined errors without risking hardware.

Accuracy on a clean benchmark like Something-Something-V2 (SSv2) is a necessary baseline and says nothing about where the model fails. For a robot in a factory or a drone in the wind, the questions are:

• Does it degrade gracefully, so downstream training can absorb it, or is there a cliff?
• Does the temporal-consistency story hold? If the features are more fragile under temporal corruption than under image noise, decisions made on that story will be wrong.
• Does moving to the 2B model buy reliability? If scaling isn’t monotonic, model selection has to be empirical and per application.

How we ran it

V-JEPA 2.1 (Mur-Labadia et al., 2026) introduces dense features through a Dense Predictive Loss, deep self-supervision, and modality-specific tokenizers. The encoder ingests video as 16-frame clips and groups consecutive frame pairs into single temporal tokens. The architecture calls this a tubelet size of 2. So a 16-frame clip becomes 8 temporal positions, each carrying a small bit of cross-frame averaging by construction. This matters later for understanding why image-noise hits harder than frame drops.

We evaluated all four released sizes: ViT-base (80M), ViT-large (300M), ViT-giant (1B), and ViT-gigantic (2B).

The setup

We ran nine controlled perturbations across ten strength levels (s ∈ [0.1, 1.0]) on 200 SSv2 validation clips to build the core robustness curves. We then measured functional tracking degradation on 30 DAVIS clips (five perturbations × five strengths × four models) to ground the representation drift in a real-world task.

Three metrics

For each clip, the encoder produces a grid of patch-level feature vectors at every temporal position. The three metrics measure different ways those features can drift between a clean clip and its perturbed version.

M1 (frame fidelity): Average cosine distance between matched patches at the same (time, position) in clean vs. perturbed. In plain terms, how much did each patch’s representation move? Low means the encoder produced almost the same feature at that location; high means the patch was reinterpreted.

M2 (temporal consistency): Cosine distance between temporal-gradient vectors (the per-patch difference feature(t+1) - feature(t)). We compare these gradient vectors clean vs. perturbed and average. In plain terms, how much did the model’s sense of motion at each location drift? This is the primary probe for V-JEPA’s core architectural claim of temporal consistency, because it isolates change between frames from the absolute frame content.

M3 (functional utility): Patch correspondence on DAVIS. We track ground-truth object regions across frames using the model’s features as the matching signal. In plain terms, if you actually used these features to track an object through the corrupted clip, how badly would tracking degrade? This is the only one of the three that measures a downstream task rather than internal feature stability.

We pre-registered six hypotheses with numerical decision rules before launch, so there was no room to tune the metric after seeing the data.

Sample sizes and seeds

200 SSv2 clips per cell sits within the standard range for video benchmarks (MVBench, CVRR-ES use 200–240 instances per task) and is comfortable for the effect sizes we observed. Bootstrap 95% CIs on per-cell M2 means are uniformly small (median ±0.015, max ±0.025 across all 200 cells). The smallest cross-model jump in the scaling story below (occlusion, +0.017) is 5.7× its CI half-width, well above the 2× threshold separating signal from noise.

We use a single random seed per cell, which follows ImageNet-C precedent (Hendrycks & Dietterich, 2019) for apples-to-apples cross-model comparison. With n = 200 clips per cell, between-clip variance dominates within-cell perturbation-realization variance. Multi-seed runs would tighten error bars but, given the size of the observed effects, would not flip any of the six hypothesis verdicts.

Calibration

To validate the M2 metric, we derived an analytical prediction for its behavior under reverse-playback inputs: if you flip the clip so the last frame plays first, the temporal-gradient vectors should reverse direction, and (under a generic assumption that gradients are not pathologically aligned across frames) the cosine distance between clean-forward and perturbed-reverse gradients should land near a specific value. For a 16-frame input at tubelet size 2, that value is approximately 1.14.

Calibration on 30 DAVIS clips with ViT-base yielded mean M2 = 1.020 (std 0.036). Across the full SSv2 sweep, means for all four models clustered tightly between 1.034 and 1.037. Both numbers sit well within the predicted [0.9, 1.4] band. The DAVIS calibration (n = 30) and the SSv2 reverse cells (n = 200/model) have non-overlapping CIs but the gap is small and likely reflects DAVIS’s smaller, hand-curated clip distribution. The metric measures what the math says it should.

1. The dense features are partitioned

M2, representational stability, predicts downstream task failure (M3) for some corruption classes and not others.

The CIs for time-axis perturbations and image-noise perturbations do not overlap. The closest gap, between occlusion’s lower bound (+0.278) and motion blur’s upper bound (+0.171), is +0.106. The two perturbation families are statistically separable at 95% confidence. The aggregate r = 0.161 (95% CI [0.126, 0.195]) is distinguishable from zero but well below the pre-registered 0.30 ambiguous threshold and 0.50 confirmation threshold.

So the features seem to have two semi-independent axes, an image-content axis that noise disturbs but tracking doesn’t lean on, and a temporal-structure axis that DAVIS-style correspondence depends on.

For Cable Mind, where self-occlusion and variable frame rates are the main stresses, M2 is a usable health check. For drone inspection, where motion blur and sensor noise dominate, a feature-level stability metric will mislead, and we have to evaluate on the task.

2. Bigger isn’t reliably better

Robustness plateaus or reverses at the largest sizes. Every Tier 1 perturbation was non-monotonic:

• The 2B “gigantic” model was less robust than the 1B “giant” on Gaussian noise (+0.038 M2 jump), motion blur (+0.050), and low-light (+0.036).
• The 1B “giant” model was less robust than the 300M “large” variant on frame drops (+0.041) and occlusion (+0.017).

All five jumps clear at least 5× their pooled CI half-width, so none of them is borderline.

One candidate mechanism is hub marginalization in deep ViTs (arXiv:2511.21635). In a Vision Transformer the [CLS] token is supposed to act as a global summary, the place where information from every patch gets aggregated. As models get deeper and better-trained, that single hub becomes less load-bearing; the patch tokens themselves start carrying distributed information rather than routing everything through one summary node. This is generally good, until a model goes too deep and crosses into an “over-communication” regime where extra layers scramble information instead of refining it. V-JEPA 2.1’s training objective (the Dense Predictive Loss) explicitly pushes against single-hub aggregation by forcing every patch token to retain local identity. If the 2B variant has crossed into the over-communication regime while the distilled 300M variant retains controlled mixing, the non-monotonic robustness pattern is exactly what hub marginalization predicts. Distillation lineage is the second mechanism. The smaller variants are distilled from the 2B teacher per the released filename convention (*_dist_vitG_*), so they may inherit teacher-level robustness without the teacher’s full capacity costs.

Scaling to 2B isn’t a shortcut to reliability. For Cable Mind, which runs on edge-class hardware, the 300M variant is often the better choice: more robust to temporal gaps, at a fraction of the compute.

3. It’s orientation-sensitive

We hypothesized that a horizontal flip would preserve M2 consistency because flipping every frame the same way changes where things are in space but doesn’t change how they move between frames. The temporal-gradient field should be unchanged in its local structure, just mirrored. The data refuted this: mean M2 = 0.914 across all models, comparable to the disruption caused by playing the video backwards.

V-JEPA 2.1’s representations are sensitive to absolute spatial orientation. Flipping space changes the feature identity of every patch, which then warps the temporal-gradient field. The model isn’t learning reflection-invariant representations out of the box. A patch on the left side of the frame produces a different feature than the same content rendered on the right side.

For drone inspection, where camera roll varies through a flight, we can’t rely on V-JEPA to handle the rotation. We either rectify frames to a gravity-aligned orientation before they reach the encoder, or train the downstream layers to absorb it.

4. Image noise hits harder than temporal corruption

V-JEPA 2.1 is framed around temporal consistency, and M2 is more disrupted by per-frame noise than by gaps in time. At strength 0.5, Gaussian noise (M2 = 0.499) is 1.54× more disruptive than frame drops (M2 = 0.324, 95% CI on the ratio [1.48, 1.60]). The dominance is not strength-specific. Gaussian dominates frame drops on at least 3 of 4 models at every one of the 10 strengths tested.

V-JEPA 2.1 looks more like a strong per-frame encoder with light temporal smoothing than a true temporal world model. The tubelet-size-2 design likely contributes: because consecutive frame pairs are pooled into a single temporal token, dropping one frame within a pair still leaves the other frame intact, and the resulting token degrades less than you’d expect. Per-frame noise, by contrast, corrupts both halves of every pair simultaneously. There’s no within-tubelet redundancy to fall back on.

What’s next

This study establishes the parametric baseline. In Part 2, we will replace these fixed perturbations with a learned adversary from Break, our adversarial search tool, still in development. The goal is to see if a small network trained to maximize degradation can find failure modes (like specific texture-sticking points) that the conditions we named missed.

Reproducibility. Code, manifests, raw shards, and the analysis notebook at github.com/poisson-labs/vjepa-stress.

Acknowledgments

V-JEPA 2.1 from Meta FAIR. Methodology inspired by the ImageNet-C work of Hendrycks & Dietterich (2019). Pre-registration discipline modeled on practices from cognitive science. Compute provided by RunPod.