LIMSSR: scoring actions when modalities go missing at training time

2026-07-17 · 6 min · multimodal · missing-modality · llm · action-quality-assessment · explainer

Action Quality Assessment (AQA) is the task of watching a video of an action — a dive, a figure-skating program, a gymnastics routine — and predicting a numeric quality score a judge would give it. The strong systems are multimodal: they read RGB frames, optical flow, and sometimes audio, because form, motion, and rhythm all carry signal.

Which is fine until a modality goes missing. And in the real world, modalities go missing not just at test time but during training — a dataset where some clips never had audio, or flow was never computed. That is the case LIMSSR (Xu, Wu, Ke, and Peng; ICML 2026 spotlight) is built for, and it is meaningfully harder than the usual setup.

Why training-time missingness is the hard case

Most missing-modality work makes a quiet assumption: you trained on complete data, learned what a "normal" audio or flow stream looks like, and only lose one at inference. Then you can reconstruct the missing stream, or distill a complete-modality teacher into an incomplete-modality student.

Take the missingness back into training and both tricks weaken. You cannot reconstruct a distribution you never fully saw, and there is no clean complete-modality teacher to distill from if the training set itself is full of holes. The score is a scalar, so a wrong imputation does not announce itself — it just quietly biases the number. To make "how much does missingness hurt" precise, AQA is graded by Spearman's rank correlation $\rho$ between predicted and ground-truth scores: 1.0 is a perfect ordering of performances, 0 is chance. It is a ranking metric, so it punishes exactly the systematic bias a bad guess introduces.

Three paradigms for incomplete-multimodal learning. (a) Reconstruction-based methods use a generative model to rebuild the missing modality before a downstream head. (b) Distillation/prior-based methods train a complete-modality teacher and distill priors into an incomplete-modality student. (c) LIMSSR tokenizes the available modality plus a text prompt naming the missing ones and feeds them to a LoRA-tuned LLM as a sequence-to-score problem. — Two older answers — reconstruct the missing modality (a), or distill a complete-modality teacher (b) — and LIMSSR's: reframe the whole thing as LLM-driven sequence-to-score reasoning (c) (Xu et al., 2026, Figure 1).

Impute in prompt space, don't reconstruct

LIMSSR's first move is to stop reconstructing. Each modality gets a frozen feature extractor and a small projection; a missing modality is not zero-filled but replaced with a special token, and the model is handed a text prompt that names which modalities are present and which are gone. The LLM is then asked to infer the missing modality's latent role from the context that remains — role inference, not feature reconstruction.

That distinction is the whole point. A zero-filled slot drags a fixed-slot fusion model toward a wrong answer; a described-absence lets a language model reason about what the gap means. Drop a modality on each side and watch the gap open:

drop a modality · watch each model reactFS1000, real ρ

present

1/3 present · naive 0.623 · LIMSSR 0.853

The naive model owns a fixed slot per modality. Remove one and it feeds zeros, so the score slides — and with audio only it falls to 0.177, barely above chance. LIMSSR turns the empty slot into a special tokenand tells the LLM, in words, which modalities are gone; the LLM infers the missing part’s latent role from what remains rather than reconstructing it, holding 0.687on the same audio-only case. (Illustrative pipeline; ρ values are the paper’s FS1000 numbers.)

The audio-only case is the tell. A naive fusion model, fed zeros for the two missing streams, collapses to $\rho = 0.177$ — barely above chance. LIMSSR, told in words that video and flow are gone and asked to reason from audio alone, holds $\rho = 0.687$ on the same split.

The pipeline

End to end: per-modality features → specific projection → prompt-guided context-aware modality imputation (special tokens + the modal-condition prompt) → an LLM-driven multidimensional representation fusion that packs everything into fusion tokens → a frozen large language model with LoRA doing the sequence-to-score reasoning → a mask-aware aggregation head that emits the score.

The LIMSSR architecture. Frozen specific feature extractors turn RGB, flow, and audio into features; a modality-missing condition and specific projection produce incomplete multimodal features; prompt-guided context-aware modality imputation and LLM-driven multidimensional representation fusion build fusion tokens; a frozen LLM with LoRA processes them under a modal-condition prompt; and a Mask-Aware Dual-Path Aggregation head combines cross-modal pattern recovery with uncertainty-calibrated reasoning to output the quality score. — The full pipeline: frozen extractors, prompt-space imputation, a LoRA-tuned LLM doing the reasoning, and the Mask-Aware Dual-Path Aggregation head (Xu et al., 2026, Figure 2).

Mask-aware dual-path aggregation: don't trust a lucky guess

Reasoning about a missing modality invites a failure mode: the model confidently hallucinates the part it cannot see. LIMSSR's aggregation head is built to suppress exactly that. It runs two paths off the same missingness mask $m$ :

Cross-modal pattern recovery — cross-attention, gated weighting, and a learnable confidence. Strong when modalities are present, shaky when they are not.
Uncertainty-calibrated reasoning — role-aware weighting and mask-aware refinement that explicitly discounts low-confidence dimensions, so it degrades gracefully.

A learnable-confidence gate blends the two. As modalities drop, the recovery path has little left to cross-attend over, its confidence falls, and the gate shifts weight onto the calibrated path — the one that already distrusts what it cannot verify. Toggle the modalities and watch the gate swing:

mask-aware dual-path aggregationgate weights, illustrative

present

1/3 present · gate 71% reason / 29% recover

When every modality is present the two paths agree and split the vote. Drop modalities and the recovery path has little left to cross-attend over, so its learnable confidencefalls and the gate shifts weight to the calibrated path — the one that already discounts what it cannot trust. That shift is the anti-hallucination mechanism: on FS1000 the full gate cuts mean-squared error from 18.18 (simple fusion) to 14.08, at ρ 0.789. (Gate dynamics illustrative.)

That shift is the anti-hallucination mechanism. On FS1000 the full gate cuts mean-squared error from 18.18 (simple fusion) to 14.08, at $\rho = 0.789$ .

Results

Across every available/missing combination, LIMSSR's predicted scores track the diagonal — the ground truth — more tightly than the multimodal-expert baselines (MoMKE, MCMoE), and it holds up in the settings where they fall apart:

Scatter plots of predicted versus true action-quality score for seven available/missing modality combinations, comparing MoMKE, MCMoE, and LIMSSR. Points for LIMSSR cluster along the diagonal in every panel, including the hardest single-modality cases, while the baselines spread further from it. — Predicted vs true score across all seven modality-availability settings — LIMSSR (right of each triplet) stays near the diagonal where MoMKE and MCMoE drift (Xu et al., 2026, Figure 5).

The starkest number is the hardest split — audio only, the two visual streams gone:

FS1000 Spearman ρ — audio only (both visual streams missing)

LIMSSR

0.69

naive fusion

0.18

00.51

With every modality present the margin is smaller but still real ( $\rho$ 0.891 vs 0.819), which is the shape you want: a method that helps most exactly where the problem is hardest, and does no harm when the data is complete.

The take

The reframing is the idea worth keeping. Missing-modality learning has mostly been treated as a reconstruction problem — rebuild the pixels or features you lost. LIMSSR treats it as a reasoning problem: describe the absence in language, let an LLM infer what the missing stream would have contributed, and gate the answer by how much you can trust it. That it works under training-time missingness — the case reconstruction and distillation both struggle with — and lifts audio-only $\rho$ from 0.177 to 0.687 is a good argument that, for messy real-world multimodal data, telling the model what it is missing beats trying to fake what it lost.

Source: "LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations" (Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng), ICML 2026 spotlight. Numbers ( $\rho$ , MSE) are the paper's reported FS1000 results; the interactive diagrams illustrate the mechanism.