EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction

Researchers introduce EigeNet, a geometry-informed deep learning framework for predicting Room Impulse Response (RIR) in spatial audio from limited observations. The model combines transformer architecture with acoustic ray tracing principles to achieve state-of-the-art performance in few-shot novel view RIR prediction and demonstrates strong sim-to-real generalization capabilities.

EigeNet addresses a fundamental challenge in immersive spatial audio: reconstructing complete acoustic environments from sparse, incomplete data. This inverse problem requires sophisticated reasoning about how sound propagates through physical spaces—a task traditionally demanding extensive measurements or computational simulations. The framework's innovation lies in combining multi-modal learning (integrating visual geometry and acoustic data) with transformer-based attention mechanisms that capture both local acoustic properties and global spatial relationships.

The research builds on growing recognition that geometric information constrains acoustic behavior. By incorporating ray tracing principles—the physical foundation of how sound travels—the model learns more meaningful representations than purely data-driven approaches. The auxiliary multi-task learning framework transforms single-waveform prediction into a richer learning problem, improving generalization across viewing angles and acoustic conditions.

For developers building spatial audio applications, this work reduces computational requirements for rendering immersive soundscapes. Rather than expensive full-scene acoustic simulations, practitioners can predict realistic RIRs from limited measurements, accelerating deployment in VR/AR environments, teleconferencing systems, and gaming engines. The sim-to-real generalization is particularly valuable, suggesting models trained on synthetic data transfer effectively to real-world recordings—a persistent challenge in applied machine learning.

The open-sourced code and checkpoints enable rapid adoption by the audio research community. Future applications likely extend beyond entertainment to architectural acoustics simulation and hearing aid design optimization. The geometry-informed modulation approach offers a reusable pattern for other physics-informed learning problems requiring multi-modal integration.

• →EigeNet uses transformer architecture with cross-view attention to predict complete room acoustic responses from sparse observations
• →Geometry-informed modulation blocks connect physical room properties to acoustic predictions, improving interpretability and generalization
• →Model achieves state-of-the-art performance on both simulated benchmarks and real-world acoustic datasets
• →Multi-task auxiliary loss framework outperforms single-target prediction approaches across different backbone architectures
• →Strong sim-to-real generalization enables practical deployment for immersive audio applications with minimal real-world data