RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview

RuleEdit is an interactive AI system that helps practitioners detect model failures and preview the impact of edits before implementation. Tested in stroke rehabilitation assessment, it increased human-AI performance by 14.16% through interpretable failure signals and prospective impact previews, though it revealed a critical local-global performance tradeoff where edits optimizing specific cases can degrade broader performance.

RuleEdit addresses a fundamental gap in AI deployment: the inability to safely inspect and modify model behavior before committing changes to production systems. The system combines two powerful mechanisms—interpretable mismatch signals that surface failure modes and prospective previews showing how edits affect performance—to create a failure-aware human-AI editing workflow. This matters because practitioners currently operate blind, unable to distinguish between reliable and unreliable AI predictions or understand consequences of interventions.

The research emerges from growing recognition that AI systems require human oversight and controllability, particularly in high-stakes domains like healthcare. Current approaches either treat models as black boxes or demand extensive retraining. RuleEdit bridges this gap through rule-guided editing, enabling domain experts to iteratively refine models without deep machine learning expertise.

The healthcare application demonstrates measurable impact: the 14.16% performance improvement reflects reduced both over-reliance (blind trust in AI) and under-reliance (excessive skepticism). More critically, prospective previews increased local performance gains from 11.50% to 36.38%, suggesting visualization of model internals significantly improves human feedback quality. This has implications for medical AI deployment, where transparency and controllability directly affect clinical adoption and patient safety.

However, the revealed local-global tradeoff represents a fundamental challenge: optimizing for specific cases risks degrading generalized performance. This finding suggests future human-AI systems must balance targeted interventions against global robustness, requiring new frameworks for managing these tensions. The work points toward AI systems that remain transparent, editable, and failure-aware throughout their operational lifetime.

• →RuleEdit increased human-AI performance by 14.16% through interpretable failure detection and rule-guided feedback mechanisms.
• →Prospective impact previews nearly tripled local performance gains (11.50% to 36.38%) by helping users author better model edits.
• →The system successfully reduced both over-reliance and under-reliance on AI predictions in clinical assessment tasks.
• →A critical local-global tradeoff emerged where edits optimizing specific cases can degrade broader model performance when deployed globally.
• →The research demonstrates that failure-aware, controllable human-AI systems require transparency mechanisms and preview capabilities for safe model modification.