Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats

Researchers have developed a framework that enables 3D Gaussian Splatting (3DGS) assets to participate in complex, physics-based simulations alongside traditional CG assets in full scenes. By translating diverse assets into a unified particle representation, the work overcomes previous limitations that restricted physics interactions to isolated, object-centric scenarios, enabling realistic two-way interactions between deformable 3DGS objects, fluids, meshes, and captured environments.

This research addresses a fundamental gap in 3D graphics and simulation technology. While 3D Gaussian Splatting has revolutionized photorealistic rendering through its efficient neural representation, it remained isolated from physics engines due to a representation mismatch. Previous attempts to integrate physics with 3DGS were limited to monolithic, single-object scenarios on simplified planes, preventing realistic interactive experiences in complex environments.

The breakthrough lies in the Representation Abstraction Framework, which converts all asset types—3DGS, meshes, fluids, and collision geometry—into a unified particle-based representation. This abstraction enables physics solvers to operate agnostically across heterogeneous assets without needing to understand each format individually. The framework then remaps computed physical results back to each asset's native representation for rendering.

For the graphics and interactive media industry, this unlocks significant potential in game development, virtual production, and architectural visualization. Teams can now leverage photorealistic 3DGS assets in fully interactive environments where objects deform realistically and respond to fluid dynamics and collisions. This eliminates the workflow friction of converting between rendering and simulation formats.

The implications extend to real-time applications where computational efficiency matters. Since 3DGS offers faster rendering than traditional mesh-based approaches, integrating it with physics simulation could enable more complex, real-time interactive experiences. Future development will likely focus on performance optimization and expanding solver compatibility to handle even more complex multi-body dynamics and constraint systems.

• →3DGS assets can now participate in full scene-level physics simulations alongside traditional CG assets and fluids
• →A unified particle abstraction framework enables different asset types to interact physically without format conversion
• →Non-rigid deformation of 3DGS objects becomes possible within a single physics pipeline
• →Static scene geometry captured from real environments can serve as collision boundaries in simulations
• →The solver-agnostic design allows compatibility with multiple physics engines and solvers