Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

Researchers present PROBE, a framework that improves how AI software engineering agents recover from failures by converting runtime telemetry into structured diagnoses and bounded recovery guidance. The system achieves 65% diagnosis accuracy and 21.8% recovery rates on previously unresolved cases, with a prototype deployed at Microsoft showing practical viability without disrupting existing workflows.

PROBE addresses a critical bottleneck in autonomous software engineering: the gap between detecting failures and recovering from them. Current systems generate error traces or feedback, but lack systematic mechanisms to translate heterogeneous runtime signals into actionable recovery steps for subsequent attempts. This research demonstrates that structured failure analysis—organizing telemetry into evidence, diagnosis, and guidance layers—significantly improves agent resilience.

The framework's design reflects lessons from operational troubleshooting. The Telemetry Layer preserves fine-grained runtime signals rather than discarding information, the Diagnosis Layer synthesizes cross-signal evidence into grounded explanations, and critically, the Guidance Gate enforces constraints: guidance must be evidence-grounded, actionable by agents, and behaviorally feasible. This prevents agents from receiving speculative or out-of-scope recommendations that waste recovery attempts.

Evaluation across three domains—repository-level repairs, enterprise workflows, and cloud service mitigation—reveals consistent improvement: PROBE outperforms baselines by 43.58 percentage points on diagnosis and 12.45 points on recovery. The Microsoft IcM deployment proves the framework integrates non-intrusively with existing systems, critical for enterprise adoption. However, the 21.8% recovery rate on unresolved cases indicates substantial room for improvement; diagnosis accuracy alone (65%) doesn't guarantee successful recovery, exposing a structural challenge in translating understanding into executable fixes.

This work matters for AI infrastructure costs. Better failure recovery reduces manual intervention and re-execution expenses, making autonomous agents economically viable at scale. As enterprises deploy more AI agents in critical workflows, systematic recovery frameworks become essential infrastructure.

• →PROBE's structured recovery framework achieves 65.37% diagnosis accuracy on previously unresolved software failures, outperforming baselines by 43.58 percentage points.
• →The framework successfully translates runtime telemetry into actionable recovery guidance only when evidence-grounded and within agent execution scope, enforcing realistic constraints.
• →Deployed at Microsoft as a non-intrusive side channel, PROBE integrates with existing workflows without modifying agent policies or toolsets, demonstrating production viability.
• →The diagnosis-recovery gap reveals that accurate failure diagnosis is necessary but insufficient—bounded, executable guidance is required for successful agent recovery.
• →21.8% recovery rate on 257 initially unresolved cases indicates significant cost savings from reduced manual intervention in enterprise software engineering workflows.