Sensorimotor World Models: Perception for Action via Inverse Dynamics

Researchers introduce Sensorimotor World Models (SMWM), a latent world model that uses inverse dynamics regularization to learn action-aligned representations from high-dimensional observations. The approach addresses representation collapse in JEPA-style models while enabling efficient planning without frozen encoders or complex regularizers, demonstrating competitive performance on control tasks.

The research tackles a fundamental challenge in machine learning: how to build world models that compress high-dimensional sensory data into useful representations for decision-making. Traditional approaches either sacrifice prediction accuracy for interpretability or suffer from representation collapse—where models learn uninformative latent spaces because their only objective is making future states easy to predict. SMWM solves this by adding inverse dynamics regularization, which forces the model to retain information about which action caused each state transition.

This breakthrough builds on growing recognition that perception and action are inseparable. Rather than optimizing for visual fidelity alone, the model learns representations shaped by what matters for control. By preserving action information in latent states, SMWM naturally focuses on controllable degrees of freedom while ignoring visual distractors like flickering lights or moving backgrounds—a significant advantage for robotic and autonomous systems.

The practical implications are substantial. The method trains end-to-end from offline, reward-free trajectories without requiring frozen encoders, exponential moving averages, or hand-crafted latent regularizers. This simplicity reduces engineering overhead and makes the approach more accessible to practitioners. The learned latent spaces are compact and interpretable, enabling stronger planning performance across diverse control tasks.

For the AI development community, SMWM represents incremental but meaningful progress toward more efficient world models. The approach could accelerate research in robotics, autonomous vehicles, and embodied AI systems where action-aligned representations are critical. However, this remains primarily a research contribution requiring further validation at scale and in real-world deployment scenarios.

• →Inverse dynamics regularization prevents representation collapse while inducing action-aligned latent representations in world models.
• →SMWM trains end-to-end without frozen encoders or complex regularizers, simplifying implementation compared to existing JEPA-style approaches.
• →The method learns compact, interpretable latent spaces optimized for control rather than visual fidelity.
• →Offline, reward-free training enables the model to learn from unlabeled trajectory data without active interaction.
• →Competitive planning performance on 2D and 3D control tasks demonstrates practical utility for robotics and autonomous systems.