Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

Researchers reveal that vision-language models (VLMs) fail to recognize when spatial questions cannot be reliably answered due to occlusion or perspective ambiguity, instead producing overconfident incorrect responses. The study introduces SpatialUncertain, a benchmark showing that current VLMs achieve only 30% accuracy under occlusion and below 10% under perspective challenges, highlighting a critical gap between answer correctness and epistemic awareness.

Vision-language models are increasingly deployed in real-world applications where spatial reasoning determines critical decisions—from robotics to autonomous systems. This research exposes a fundamental vulnerability: VLMs conflate visual input with complete information about 3D environments. When objects are occluded or perspective creates ambiguity, models continue generating answers with high confidence rather than signaling uncertainty, creating false confidence in unreliable outputs.

The research emerges from growing recognition that benchmark design shapes model behavior. Traditional spatial reasoning evaluations reward answer correctness without penalizing overconfidence, incentivizing models to attempt answering even when evidence is insufficient. This mirrors broader AI safety concerns about calibrated uncertainty in language models, but applies specifically to multimodal reasoning where incomplete visual observation is unavoidable in physical deployments.

The implications extend beyond academic concern. Robots, drones, and autonomous systems relying on VLM spatial understanding could make catastrophic mistakes when models confidently misinterpret occluded or ambiguous scenes. The finding that models perform near-random chance when identifying which additional viewpoints would resolve ambiguity suggests they lack meaningful reasoning about information gaps—they guess rather than reason about epistemic needs.

Moving forward, the field faces pressure to develop VLMs that maintain epistemic awareness proportional to evidence quality. This requires architectural changes enabling abstention and genuine uncertainty quantification, not just confidence calibration. Organizations building safety-critical systems using VLMs should prioritize models demonstrating these capabilities, as current frontier models remain untrustworthy for applications where false confidence creates liability.

• →VLMs show overconfident answering on spatial tasks with incomplete information, achieving only 30% accuracy under occlusion and below 10% under perspective ambiguity.
• →Current benchmarks reward answer correctness without measuring whether models recognize when questions cannot be reliably answered, creating misaligned incentives.
• →Models fail to identify which additional viewpoints would resolve perspective ambiguity, performing near random chance on this critical epistemic task.
• →The gap between answer production and uncertainty awareness poses real safety risks for robotics and autonomous systems relying on VLM spatial reasoning.
• →Future VLM development must prioritize epistemic awareness and abstention capabilities alongside task accuracy to enable trustworthy deployment in real-world environments.