Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security

Researchers introduced Runtime Skill Audit (RSA), a dynamic analysis method that detects malicious behavior in LLM agent skills by testing them under targeted runtime conditions rather than relying on static code review. RSA achieved 90% accuracy in identifying harmful skills and maintained effectiveness against evolving attacks where static methods failed, addressing a critical security gap in agent-based AI systems.

The emergence of LLM agents that leverage reusable skills represents a significant architectural shift in AI systems, but introduces substantial security vulnerabilities that traditional static analysis cannot adequately address. Malicious skills can remain hidden in code and documentation while manifesting harm only under specific execution contexts—a problem that purely code-based vetting mechanisms systematically fail to detect. RSA addresses this gap by simulating actual runtime behavior, profiling risk-relevant interfaces, and examining execution traces to assign security labels based on observable actions rather than code inspection.

This research reflects a broader maturation challenge in AI infrastructure security. As LLM agents become more autonomous and interconnected with external tools and persistent state, the attack surface expands dramatically. Traditional security practices designed for static software environments prove inadequate when behavior emerges dynamically from agent-tool interactions. The 13 percentage-point accuracy improvement over static baselines, combined with RSA's resilience against adaptive attacks, demonstrates that dynamic analysis is becoming essential rather than supplementary.

For developers and enterprises deploying agent-based systems, this work signals that security practices must evolve beyond code review toward runtime monitoring. The stark contrast between RSA's consistent detection (19-20 malicious skills) and static methods' collapse after one to two rounds of attack evolution indicates that any production agent system relying exclusively on static vetting faces escalating risk. Organizations building agent ecosystems—including those integrating blockchain interactions, DeFi protocols, or other high-stakes operations—should prioritize dynamic analysis frameworks. The research underscores that as agents gain autonomy, observability of actual behavior becomes the critical control mechanism.

• →Runtime Skill Audit detects malicious agent behaviors that static code analysis misses by monitoring actual execution under targeted conditions
• →RSA achieved 90% accuracy with 88% true positive rate, outperforming static baselines by 13 percentage points
• →Static security detectors fail catastrophically against adaptive attacks while RSA maintained 95-100% detection rates across multiple attack rounds
• →Agent skills can appear benign in documentation while exhibiting harmful behavior only through specific multi-step tool interactions and context combinations
• →Dynamic analysis is becoming essential for securing autonomous agent systems that interact with external tools and persistent state