Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Researchers developed a three-stage pipeline to automatically extract skill libraries from computer-using agent interaction data, achieving high readability (95% purity on labeled benchmarks) but failing to improve downstream policy performance across domains. The study reveals that while trajectory mining can expose interpretable skill structure, current technical limitations prevent reliable cross-domain transfer improvements.

This research addresses a critical challenge in AI agent development: whether automatically mined skill libraries from interaction data can enhance policy performance while maintaining interpretability. The three-stage approach—segmenting GUI trajectories, clustering segments into skills, and training skill-aware policies—represents a systematic attempt to bridge the gap between explainability and functionality in computer-using agents.

The work builds on growing recognition that explicit skill representations improve agent inspectability, a key concern for deploying autonomous systems in complex environments like web browsing and software interaction. Prior efforts focused on manual skill curation or limited automation; this study scales skill extraction to interaction data directly.

However, the results reveal a sobering gap: while five of eight mined clusters achieved 95% purity against established workflow labels, the method failed to deliver meaningful policy improvements. GRPO performance increased only marginally (18.5% to 20.5% on InteraSkill) and remained essentially flat on BrowseComp+, underperforming simple frequency baselines on source metrics. This disconnect between readability and transfer effectiveness indicates the current pipeline lacks sufficient sophistication in three critical areas: boundary detection for skill segments, segment representation that captures order-dependent nuances, and offline reward models that generalize across domains.

For the AI agent research community, this diagnostic study prevents overconfidence in human-readable clusters as a guarantee of improved performance. It highlights that inspectability and transferability represent distinct engineering challenges requiring separate solutions, not downstream consequences of the same approach.

• →Automatically mined skill clusters achieve high interpretability (95% purity) but fail to improve downstream policy performance on multiple benchmarks.
• →The three-stage pipeline successfully extracts readable skill structure from interaction trajectories, validating the core mining methodology despite performance limitations.
• →Current boundary detection, segment representation, and offline reward models prove insufficient for reliable cross-domain policy transfer.
• →Readability of skill libraries does not guarantee their utility for training better agents, indicating these are separate engineering objectives.
• →The research provides diagnostic insights for future work on skill-aware agent architectures and trajectory mining techniques.