Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting

Ilov3Splat introduces a framework for understanding 3D scenes using natural language by combining 3D Gaussian Splatting with CLIP features and SAM masks. The method achieves better cross-view consistency and instance-level reasoning than prior approaches, enabling object identification without manual annotation.

Ilov3Splat represents a meaningful advancement in 3D scene understanding by addressing fundamental limitations in how AI systems perceive and label spatial environments. Previous approaches relied on 2D rendering-based matching or point-level semantic association, which created inconsistencies when viewing objects from different angles and failed to maintain coherent object-level reasoning. The framework solves this by jointly optimizing geometric and semantic representations, using multi-resolution hash embedding to encode language-aligned CLIP features throughout 3D space.

This work builds on the broader trend of combining foundational vision models with 3D representations. CLIP's language understanding and SAM's segmentation capabilities are increasingly being integrated into 3D pipelines to enable more intuitive, annotation-free scene understanding. The use of contrastive learning over SAM masks allows the system to distinguish fine-grained object differences across viewpoints, a capability essential for robotic applications and spatial AI systems.

The implications extend beyond academic interest. For robotics, autonomous systems, and spatial computing applications, language-driven 3D understanding eliminates expensive manual labeling workflows. Companies developing embodied AI systems—from warehouse robots to autonomous vehicles—benefit from methods that convert natural language queries into precise 3D object identification without requiring task-specific training data.

The open-vocabulary nature is particularly significant, as it enables systems to recognize and interact with arbitrary objects rather than predefined categories. Future work likely involves scaling this to real-time applications and testing robustness across diverse environments and lighting conditions. The project's public availability suggests academic and industry adoption will follow, potentially influencing how 3D scene understanding is approached in production systems.

• →Ilov3Splat combines 3D Gaussian Splatting with CLIP features to enable language-driven 3D scene understanding without manual annotations.
• →The method achieves superior cross-view consistency and instance-level reasoning compared to previous rendering-based approaches.
• →Multi-resolution hash embedding efficiently encodes dense semantic features throughout 3D space for coherent object grounding.
• →The framework identifies arbitrary objects via natural language queries, eliminating the need for category-specific training data.
• →Results demonstrate improved performance in both object selection and instance segmentation tasks on standard benchmarks.