SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

Researchers introduce SeePhys Pro, a benchmark revealing that advanced AI models significantly degrade in physics reasoning when visual information replaces text, with visual grounding as the primary failure point. The study further demonstrates that multimodal reinforcement learning improvements can stem from non-visual textual cues rather than genuine visual understanding, challenging current evaluation methodologies.

SeePhys Pro addresses a critical blind spot in multimodal AI evaluation: the assumption that models reasoning effectively with text will maintain performance when the same information appears as diagrams. This research exposes representation-fragility in frontier models, showing average performance decline as information modality shifts from language to images. The benchmark's progressive visual variants create a controlled testing environment absent in existing vision benchmarks that evaluate single input forms.

The findings reflect broader concerns about whether multimodal systems genuinely integrate visual reasoning or exploit superficial correlations. The blind-training experiments—where models improve on unmasked validation sets despite masked training images—reveal that performance gains often correlate with residual textual and distributional artifacts rather than valid visual evidence. This diagnostic approach distinguishes between apparent improvement and substantive capability gains.

For AI development, these results mandate methodological shifts in evaluation. Current metrics relying on final-answer accuracy mask underlying fragilities in modality transfer robustness. The research suggests developers must implement targeted diagnostics testing whether improvements depend on task-critical visual evidence, not incidental cues. This has implications for safety and reliability in real-world applications where visual information varies or degrades.

Future multimodal systems require architecture designs prioritizing true cross-modal reasoning rather than unimodal learning disguised as multimodal. The work establishes evaluation standards that distinguish shallow performance from genuine representation-invariance, essential for deploying trustworthy AI systems in physics reasoning, scientific applications, and domains requiring robust visual understanding.

• →Frontier AI models show significant performance degradation when critical information transfers from text to visual format, indicating non-representation-invariant reasoning.
• →Visual variable grounding emerges as the most critical bottleneck in multimodal physics reasoning tasks.
• →Blind training experiments reveal performance improvements can arise from residual textual and distributional cues rather than genuine visual understanding.
• →Current evaluation metrics based solely on final-answer accuracy fail to capture robustness under modality transfer or dependency on task-critical visual evidence.
• →Researchers propose diagnostic controls and format-saturation testing as essential methodologies for validating true multimodal reasoning capabilities.