MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models

Researchers introduce MatterDoor, a method enabling autonomous robots to infer hidden room structure and semantics from doorway-occluded views using pretrained generative vision models without task-specific training. The approach combines VLM-guided outpainting, depth estimation, and semantic segmentation to generate 3D hypotheses of unobserved spaces, evaluated on a new Matterport3D-derived benchmark for robot navigation and object-reaching tasks.

MatterDoor addresses a fundamental challenge in autonomous robotics: reasoning about partially observable environments where doorways and walls occlude critical information needed for safe navigation and task completion. The research demonstrates that off-the-shelf pretrained generative models can function as effective zero-shot priors for robot planning, eliminating the need for problem-specific fine-tuning. This finding has significant implications for robotics deployment, as it reduces engineering overhead and enables more flexible, generalizable systems.

The technical approach leverages recent advances in vision-language models and generative AI by combining multiple complementary techniques. VLM-guided outpainting generates plausible continuations of hidden scenes, monocular depth estimation provides geometric structure, and semantic segmentation adds task-relevant labels. The resulting pipeline produces probabilistic 3D point cloud hypotheses that robots can query to estimate object locations and occupancy in hidden regions.

The introduction of the MatterDoor benchmark, derived from Matterport3D's extensive indoor scene dataset, provides a valuable evaluation framework for the community. This addresses a gap where robotics research often lacks standardized benchmarks for occlusion-aware reasoning. The simulated Stretch robot experiments validate practical utility, suggesting these priors meaningfully improve planning in real-world scenarios.

For the broader AI and robotics industry, this work exemplifies how foundation models trained on diverse internet data can transfer effectively to specialized domains. As generative models become more capable and efficient, similar approaches may unlock new capabilities across robotics, augmented reality, and spatial reasoning applications. The zero-shot nature of the method positions it as a potentially scalable solution adaptable to diverse environments and robot morphologies.

• →Pretrained generative vision models can infer hidden room structure and semantics from occluded doorway views without task-specific fine-tuning
• →The pipeline combines VLM-guided outpainting, depth estimation, and semantic segmentation to generate probabilistic 3D hypotheses of unobserved spaces
• →MatterDoor benchmark provides standardized evaluation for doorway-occluded scene understanding on Matterport3D-derived indoor environments
• →Simulated Stretch robot experiments demonstrate practical utility for object-reaching tasks in partially observable environments
• →Foundation model transfer learning reduces engineering overhead compared to traditional robotics approaches requiring specialized training