Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion

Researchers introduce Real2SAM2Real, a framework that enhances Video Diffusion Models by incorporating explicit 3D geometric caches extracted from SAM3D models, enabling more precise control over camera movements and scene dynamics while maintaining structural consistency in complex occlusions and high-motion scenarios.

Real2SAM2Real addresses a fundamental limitation in video synthesis technology: existing Video Diffusion Models generate unseen regions based purely on learned priors, which often fails catastrophically during complex scene dynamics, severe occlusions, and large camera movements. The framework introduces a novel approach by decoupling geometry from appearance through 3D lifting models, creating what researchers term a 'generative 3D cache' that serves as structural scaffolding for the diffusion process. This represents a significant methodological shift from purely implicit generation toward hybrid explicit-implicit approaches.

The technical innovation centers on leveraging SAM3D to extract complete 3D volumes of foreground entities rather than reconstructing only visible surfaces, addressing perspective ambiguities that plague traditional approaches. The researchers developed a Soft Spatial-Aligned Injection mechanism that preserves pre-trained model capabilities while integrating 3D guidance, alongside a minimally invasive fine-tuning strategy. The use of masked normal maps as a cross-modal bridge enables efficient data curation without requiring extensive 3D annotations, making the approach more scalable.

For the AI research community, this work demonstrates the complementary value of combining classical 3D graphics principles with modern deep learning, potentially influencing how future generative models handle spatial control. The ability to decouple camera trajectories from multi-entity motions opens new possibilities for controllable video generation in animation, film production, and virtual reality applications. The framework's robustness under severe occlusions suggests practical applications in scenarios where traditional structure-from-motion fails. Industry practitioners developing video generation tools should monitor this research direction, as it validates that hybrid geometric-learning approaches may outperform end-to-end learned solutions for spatially-constrained generation tasks.

• →Combines 3D lifting models with video diffusion to create explicit geometric scaffolding for improved structural stability
• →Enables decoupled control over both camera trajectories and multi-entity motions simultaneously
• →Resolves structural collapse during high-dynamic movements by eliminating over-reliance on implicit diffusion priors
• →Uses masked normal maps to create efficient data pipelines without extensive 3D annotation requirements
• →Demonstrates superior performance under severe occlusions and large camera shifts compared to baseline approaches