ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

Researchers introduce ViCuR, a visual-grounded distillation framework that improves multimodal AI reasoning by using recoverable visual cues instead of answer-dependent privileges. The approach achieves consistent performance gains across seven benchmarks with Qwen3-VL models by eliminating train-test mismatches that encourage shortcut learning rather than genuine visual understanding.

ViCuR addresses a fundamental problem in multimodal machine learning: the performance gap between training and deployment when teachers access privileged information unavailable at inference time. Traditional on-policy distillation frameworks use answer-side privileges like reference answers or rationales, creating scenarios where students learn to imitate shortcuts rather than develop robust visual reasoning capabilities. This architectural misalignment between training supervision and inference constraints has long plagued multimodal systems, particularly in vision-language models.

The innovation centers on replacing inaccessible answer-side privileges with visual cues—query-relevant evidence extracted from the same input images available during inference. By grounding privilege in recoverable visual information, ViCuR ensures that anything the teacher leverages can theoretically be learned and accessed by the student. The lightweight cue recovery module uses sink-token cross-attention during prefill to aggregate visual evidence without altering the inference interface, maintaining deployment simplicity while improving training dynamics.

Empirical results demonstrate meaningful improvements: +1.19 to +1.24 overall performance gains across seven benchmarks compared to answer-based self-distillation baselines, with additional gains when combined with stronger teachers. The out-of-domain performance improvements at the 8B scale suggest the approach builds more generalizable reasoning rather than memorization patterns. This research validates that privilege design matters as much as teacher model strength—a principle likely applicable beyond vision-language tasks to any multimodal reasoning domain. The work contributes to making distillation frameworks more theoretically sound and practically deployable.

• →ViCuR replaces answer-side privilege with visual cues to eliminate train-test mismatches in multimodal distillation
• →Lightweight cue recovery module uses sink-token cross-attention without changing inference interfaces
• →Achieves +1.19 to +1.24 performance gains across seven benchmarks with Qwen3-VL models
• →Demonstrates consistent out-of-domain improvements, indicating learned visual reasoning rather than shortcuts
• →Framework shows privilege design is as critical as teacher strength in on-policy distillation