RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

RayDer introduces a unified transformer architecture that consolidates camera estimation, scene reconstruction, and rendering into a single model for self-supervised novel view synthesis from real-world video. The system achieves clean power-law scaling with data and compute while maintaining competitive performance with supervised approaches, addressing a key scalability challenge in 3D vision.

RayDer represents a significant architectural simplification in novel view synthesis, moving away from brittle multi-network systems toward a consolidated single-model design. This shift addresses a fundamental challenge in 3D computer vision: scaling self-supervised learning on realistic, unconstrained video data. The technical innovation lies in treating dynamic content as a minimal nuisance factor rather than attempting full 4D reconstruction, allowing the model to leverage temporal variation purely as supervision while maintaining static-scene synthesis as the core task.

The research builds on years of attempts to scale vision models beyond controlled datasets. Previous approaches struggled with the unpredictable scaling behavior of multi-component systems and the instability of training on real-world video. RayDer's unified architecture eliminates these failure modes by consolidating different computational functions into a single backbone, creating a well-posed scaling problem that follows clean power laws—a hallmark of properly designed machine learning systems.

For the broader AI industry, this work demonstrates that architectural elegance and constraint selection drive scaling efficiency more than raw model capacity. The competitive zero-shot performance against supervised methods suggests that self-supervised learning from video, when properly structured, can match or exceed supervised training. This has implications for 3D vision systems used in robotics, autonomous systems, and digital content creation, where labeled 3D data remains expensive to acquire.

The clean power-law scaling behavior indicates RayDer will benefit predictably from increased compute and data—a property crucial for production systems. Future work likely focuses on extending this approach to dynamic scenes and real-time applications, expanding the practical utility of scalable 3D understanding.

• →RayDer unifies camera estimation, scene reconstruction, and rendering into a single transformer, eliminating brittle multi-network scaling issues
• →The system achieves clean power-law scaling with data and compute, enabling predictable performance improvements
• →Dynamic content is treated as supervision rather than reconstructed, keeping static-scene synthesis as the core task
• →Zero-shot performance is competitive with state-of-the-art supervised approaches across multiple benchmarks
• →Self-supervised learning from real-world video becomes viable at scale through architectural simplification and proper constraint selection