Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models

Researchers introduce GLAM (Grounded Latent-Action World Models), a machine learning framework that learns unified action representations across heterogeneous data sources with different action spaces and missing labels. The approach achieves 48% average improvement in task success rates for robotic manipulation tasks by grounding latent actions in environmental prediction rather than relying on hand-engineered alignment techniques.

GLAM addresses a fundamental challenge in imitation learning: the scarcity and inconsistency of high-quality demonstration data. Traditional approaches struggle when combining data from multiple sources with different action formats, requiring expensive manual alignment. This research pivots toward a principle-driven solution where actions derive meaning from their environmental effects rather than their surface representations.

The underlying innovation leverages world models—generative systems that predict future observations—as a grounding mechanism. By forcing action representations to predict consistent future states across heterogeneous sources, the framework naturally discovers aligned action spaces without human intervention. This elegantly transforms a data heterogeneity problem into a prediction consistency problem, which machine learning systems handle effectively.

For the robotics and embodied AI sectors, this breakthrough reduces dependency on painstakingly curated datasets. Practitioners can now incorporate YouTube videos, human demonstrations, and synthetic data simultaneously, expanding the information available for policy training. The 48% success rate improvement demonstrates tangible gains even in data-scarce settings, addressing a persistent bottleneck in robot learning deployment.

The approach's implications extend beyond robotics. Any domain requiring behavioral learning from mixed-quality sources—autonomous vehicles, game AI, or industrial automation—could benefit from similar latent-action grounding principles. The open release of code and results supports reproducibility and broader adoption. Future work may explore scaling to more complex action spaces, longer-horizon tasks, and cross-embodiment transfer learning, potentially accelerating progress toward more generalizable autonomous systems.

• →GLAM learns unified action representations across heterogeneous data sources by grounding actions in future observation prediction rather than manual alignment
• →Framework achieves 48% average improvement in robotic task success rates compared to behavioral cloning baselines in data-scarce settings
• →Method eliminates need for hand-engineered alignment techniques between different action spaces and labeled data sources
• →Approach enables learning from abundant but messy data including unlabeled demonstrations and cross-source information
• →Open-source release facilitates adoption across robotics and embodied AI communities