Evidence Over Plans: Online Trajectory Verification for Skill Distillation

Researchers introduce SPARK, a framework that verifies AI agent skills through direct environment interaction rather than relying on pre-written plans. The Posterior Distillation Index (PDI) metric ensures skills are grounded in actual task evidence, producing student models that match or exceed human-written skills while reducing inference costs by up to 1,000x.

This research addresses a fundamental challenge in AI skill transfer: the gap between theoretical planning and practical execution. Traditional skill distillation methods depend heavily on preference logs and prior plans, often failing to improve task performance meaningfully. SPARK reframes this problem by establishing that robust skills must be posterior-based—derived from empirical evidence of actual environment interaction rather than abstract procedural knowledge. This distinction matters because it shifts focus from what agents are supposed to do to what they actually accomplish.

The Posterior Distillation Index represents a meaningful contribution to AI verification systems. By creating a trajectory-level metric that quantifies how well distilled skills align with environment evidence, researchers established a measurable standard for skill quality that goes beyond subjective assessment. The framework preserves complete execution records, enabling comprehensive analysis of agent behavior across diverse scenarios.

The practical implications are substantial for AI development and deployment. Across 86 tested tasks, SPARK-generated skills consistently outperformed no-skill baselines and matched or surpassed human-written skills on smaller, more efficient student models. The inference cost reduction—up to 1,000x cheaper than teacher models—directly addresses scalability concerns in production AI systems. This efficiency gain matters for organizations seeking cost-effective AI deployment without sacrificing performance.

The transferability of PDI-guided skills suggests broader applications beyond the tested 86 tasks. As AI systems become more prevalent in production environments, methods for generating reliable, efficient, and verifiable skills become increasingly critical. The open-source release signals the authors' confidence in the approach and invites community validation and extension.

• →SPARK generates environment-verified skills grounded in actual task execution rather than prior plans, addressing a fundamental bottleneck in skill distillation
• →The Posterior Distillation Index provides a quantifiable metric for assessing skill quality based on trajectory-level evidence from environment interaction
• →SPARK-generated skills achieve up to 1,000x inference cost reduction compared to teacher models while maintaining or exceeding performance on human-written baselines
• →The framework successfully scaled to 86 runnable tasks, demonstrating practical viability across diverse scenarios
• →Open-source code release enables community validation and extension of the posterior-based skill distillation methodology