ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents

Researchers introduce ProcCtrlBench, a new evaluation framework for LLM coding agents that measures execution-process quality rather than just final outcomes. The benchmark identifies 11 types of execution defects and introduces 'control preservation' metrics to assess whether AI agents maintain interpretability, interruptibility, and reversibility during code execution.

ProcCtrlBench addresses a critical gap in AI agent evaluation methodology. Current benchmarks treat LLM coding agents as black boxes, measuring only whether they produce correct final code. This approach obscures intermediate failures, inefficient execution paths, and loss of human oversight during the coding process. The research introduces a structured ontology of execution defects and standardizes heterogeneous agent logs into comparable formats, enabling meaningful comparison across different systems.

This work reflects growing maturity in the AI agent evaluation space. As LLMs move from isolated tasks to autonomous agents controlling systems, execution transparency becomes essential for enterprise adoption. The 'control preservation' framework directly addresses production concerns: can humans understand what the agent did, interrupt problematic actions, reverse mistakes, or regain control when needed. Testing across AndroidBench, TerminalBench, and SWE-bench-Verified demonstrates practical applicability.

The findings reveal that conventional outcome-based metrics miss significant execution-quality differences. An agent producing correct code through chaotic execution paths poses greater deployment risk than one following coherent, reversible steps. For developers building AI-assisted coding tools, ProcCtrlBench provides actionable diagnostics beyond binary success/failure metrics. For enterprises evaluating coding agents, execution transparency directly impacts liability, debuggability, and operational safety.

The benchmark's reliable instantiation and stable semantics suggest it could become standardized in AI agent evaluation. Future work likely extends control preservation concepts beyond coding domains to robotics, data processing, and financial systems—anywhere autonomous agents interact with critical infrastructure.

• →ProcCtrlBench measures execution process quality through 11 standardized defect types, not just final coding outcomes
• →Control preservation metrics assess whether agents maintain interpretability, interruptibility, and human oversight during execution
• →Testing reveals meaningful execution-quality differences missed by conventional outcome-based benchmarks
• →Standardized trajectory representation enables fair comparison across heterogeneous LLM coding agents
• →Framework addresses enterprise adoption requirements for transparency and reversibility in autonomous AI systems