Turning Video Models into Generalist Robot Policies

Researchers present VERA, a decoupled approach to robot control that separates video prediction from action execution using inverse dynamics models. Rather than fine-tuning video models with action labels, the method keeps the video planner unchanged and trains embodiment-specific models to translate predicted frames into robot actions, enabling zero-shot cross-embodiment generalization.

The research addresses a fundamental challenge in robot learning: how to leverage powerful video generative models without requiring task-specific retraining. Traditional approaches fine-tune video models end-to-end with action supervision, creating rigid systems tied to specific robot embodiments. VERA decouples this process, treating the video model as a frozen planning component and introducing an embodiment-specific inverse dynamics model as the translation layer. This architectural choice mirrors successful patterns in machine learning where modular designs outperform monolithic ones.

The approach builds on the recent trend of using large-scale video models as robot foundations, but challenges the assumption that joint training is necessary. By designing the inverse dynamics model around robot Jacobians—the mathematical relationship between joint velocities and end-effector motion—the team creates a more physically-grounded translation mechanism. The results demonstrate both data efficiency and scalability to high-dimensional action spaces, with successful experiments ranging from 7-DoF Panda arm manipulation to 16-DoF dexterous hand control.

For robotics developers and AI companies, this work signals that video models may prove more valuable than previously thought by remaining general-purpose components rather than task-specific black boxes. The ability to pair a single video model with multiple embodiment-specific IDMs reduces computational overhead and enables faster adaptation to new robot platforms. This modularity could accelerate development cycles in robotics research and commercial applications.

Future research should explore how well this decoupled approach scales to real-world deployment with limited training data, whether the video models can handle distribution shift across environments, and whether the Jacobian-based IDM design generalizes across fundamentally different morphologies.

• →Decoupling video planning from action execution enables reusing the same video model across different robot embodiments without retraining.
• →Inverse dynamics models based on robot Jacobians provide a physically-grounded, data-efficient bridge between visual predictions and robot commands.
• →VERA achieves zero-shot cross-embodiment performance, successfully controlling arms and hands with a single video planner paired with different IDMs.
• →The modular architecture reduces computational costs and allows video models to remain embodiment-agnostic, improving generalization.
• →Results suggest decoupled planning is a viable alternative to end-to-end fine-tuning for building generalizable robot control policies.