From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

A comprehensive survey examines how human videos can be leveraged to train Vision-Language-Action (VLA) models for robot manipulation, addressing the limitation that robot demonstrations are expensive and embodiment-specific. The research categorizes four approaches for extracting actionable knowledge from human videos and identifies critical open challenges in video structuring, embodiment transfer, and real-world evaluation.

This survey addresses a fundamental bottleneck in embodied AI development: the scarcity and cost of robot-specific training data. While VLA models have demonstrated impressive generalization capabilities, their scaling has been constrained by reliance on expensive robot demonstrations that lack diversity and transferability across different hardware platforms. Human videos offer a vastly larger data source with naturally rich interactions and semantic content, but leveraging them requires solving embodiment and viewpoint mismatches.

The research categorizes existing solutions into four distinct paradigms: latent action representations that capture motion abstractions, predictive world models for temporal understanding, 2D image-plane supervision for visual learning, and 3D geometric reconstruction for spatial reasoning. This taxonomy reveals the diverse technical approaches researchers employ to bridge human-to-robot knowledge transfer.

For the robotics and AI industry, this survey signals a shift toward more scalable training methodologies that decouple learning from expensive hardware infrastructure. Companies developing embodied AI systems could dramatically reduce training costs by leveraging internet-scale video datasets, accelerating development timelines and reducing barriers to entry for smaller organizations. This democratization of embodied AI training aligns with broader trends in foundation models and transfer learning.

The survey's identification of three critical open challenges—episode structuring, action grounding, and evaluation protocols—maps concrete research directions. Success in these areas will directly determine whether human-video-based training can match or exceed robot-demonstration performance in real-world deployment scenarios. The field is approaching a maturation point where data scaling and model architecture improvements may yield significant leaps in manipulation capabilities.

• →Human videos offer abundant, diverse data for training robot control models but require solving embodiment and viewpoint transfer challenges.
• →Four distinct approaches exist for extracting robot-relevant knowledge from human videos: latent actions, world models, 2D supervision, and 3D reconstruction.
• →Scaling embodied AI through human video data could reduce training costs and democratize access to robot learning capabilities.
• →Critical open challenges include structuring unstructured videos, grounding supervision to robot-executable actions, and designing evaluation protocols for real-world transfer.
• →Success in human-to-robot knowledge transfer could accelerate development of generalizable manipulation systems across diverse hardware embodiments.