Teach-and-Repeat: Accurately Extracting Operational Knowledge from Mobile Screen Demonstrations to Empower GUI Agents

Researchers introduce Teach VLM, a vision-language model that extracts operational knowledge from mobile screen demonstrations to create interpretable instructions for GUI automation agents. The system uses a novel Teach-and-Repeat paradigm where extracted task procedures guide downstream execution agents, achieving state-of-the-art performance in operation semantics prediction and improving task success rates in Android environments.

This research addresses a critical limitation in current AI systems: the gap between perceiving visual information and understanding actionable procedures on mobile interfaces. While vision-language models have advanced significantly in static image understanding, translating dynamic screen interactions into executable, human-readable instructions remains challenging. Teach VLM solves this by extracting operation-related keyframes from demonstration videos and converting them into structured natural-language commands describing actions, UI targets, arguments, and execution sequences.

The motivation stems from the fragmented mobile ecosystem where diverse UI designs across applications prevent generic models from accurately inferring operations. By developing a data flywheel for scalable training data acquisition and introducing a Chinese Mobile Screen Teach Benchmark, the researchers addressed data scarcity—a persistent obstacle in specialized AI development. This approach parallels broader trends in AI where task-specific fine-tuning and curated benchmarks drive significant performance improvements.

The practical implications extend to mobile automation, accessibility tools, and intelligent agent systems. Organizations automating repetitive mobile tasks could benefit from procedurally-guided agents rather than brittle rule-based systems. The paradigm's interpretability—generating human-readable instructions rather than black-box actions—enables better debugging and trust in autonomous systems.

Future developments depend on scaling the data flywheel beyond Chinese interfaces and integrating Teach VLM with more sophisticated execution agents. Cross-language and cross-platform generalization remain open challenges. As mobile AI agents become more capable, models that reliably extract procedural knowledge from demonstrations could become foundational infrastructure for enterprise automation and intelligent assistants.

• →Teach VLM converts mobile screen trajectories into interpretable operational knowledge through keyframe extraction from demonstration videos.
• →The Teach-and-Repeat paradigm uses generated instructions as procedural references to guide downstream GUI execution agents.
• →A systematic data flywheel addresses the scarcity of aligned training data for mobile screen understanding tasks.
• →State-of-the-art results on operation semantics prediction and consistent task success rate improvements in Android environments validate the approach.
• →The model's interpretability enables better debugging and trust compared to black-box automation systems.