PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions

PhyGenHOI is a novel AI framework that generates physically accurate 4D dynamic scenes of humans interacting with objects based on text prompts. The system combines generative human motion models with physics-based object simulation using 3D Gaussian Splats, enabling realistic interactions like punching or kicking with proper momentum transfer and contact dynamics.

PhyGenHOI addresses a significant gap in generative AI by tackling the challenge of creating physically plausible interactions between humans and objects in 4D space. Traditional generative models struggle with physics consistency when synthesizing dynamic scenes, often producing unrealistic contact dynamics or momentum transfer. This work bridges that gap by coupling a Motion Diffusion Model that generates human actions with Material Point Method physics simulation for object behavior, using 3D Gaussian Splats as a unified differentiable representation.

The framework's innovation lies in its three supervision mechanisms: temporal synchronization between motion and object interception, contact-driven re-simulation ensuring momentum conservation upon impact, and video-based priors that enhance contact realism. This represents an important step forward in multimodal AI systems that understand and respect physical laws while generating creative content.

For AI researchers and developers, this work has implications for simulation, game development, robotics training, and digital animation. The ability to generate physically consistent human-object interactions from text could accelerate content creation pipelines and improve training data generation for embodied AI systems. The approach demonstrates how explicit physics constraints can enhance generative models, a principle applicable across various synthesis tasks.

Looking ahead, the integration of physics-aware generation with large-scale generative models could become foundational for metaverse applications, digital humans, and robotic simulation. The technique's scalability to more complex multi-agent scenarios and diverse material properties remains a key area for future development.

• →PhyGenHOI combines generative human motion with physics-based object simulation for realistic 4D human-object interactions
• →The framework uses three coupled mechanisms to ensure physically consistent contact dynamics and momentum transfer
• →3D Gaussian Splats serve as a unified differentiable representation for both human and object agents
• →The approach demonstrates that explicit physics constraints enhance generative model outputs for interactive scenes
• →Applications span digital content creation, robotics training, game development, and embodied AI systems