Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents

Researchers propose FeasiGen, a framework for automatically generating infeasible task benchmarks to evaluate whether AI agents recognize when tasks cannot be completed with available tools. Testing across nine models reveals critical weaknesses, with agents continuing execution on impossible tasks up to 73.9% of the time, though multi-agent architectures show improved performance.

Current tool-using AI agents face a significant limitation: they often fail to recognize when assigned tasks are impossible to complete, wasting computational resources through prolonged execution chains. This research addresses a practical gap between agent capability assessment and real-world deployment constraints. The FeasiGen pipeline systematically constructs impossible tasks by identifying and masking critical tools from successful execution traces, creating a rigorous evaluation methodology with 94% human-verified accuracy.

The broader context involves the growing complexity of autonomous agent systems, which increasingly operate under resource constraints and real-time requirements. As agents handle more critical applications—from data processing to financial analysis—their ability to gracefully fail becomes as important as their ability to succeed. Early termination on infeasible tasks directly reduces computational cost and improves user experience.

The evaluation results are sobering for the AI development community. A false continue rate reaching 73.9% demonstrates that most current models lack robust feasibility awareness, essentially operating without failure detection mechanisms. This represents a fundamental capability gap that developers must address before deploying agents in production environments where resource efficiency is paramount.

The observation that multi-agent architectures substantially reduce erroneous execution suggests architectural solutions exist. Going forward, developers should prioritize feasibility assessment mechanisms and multi-agent design patterns. The framework also enables future research into training agents with explicit feasibility detection objectives, potentially reshaping how autonomous systems are evaluated and deployed.

• →Most AI agents continue executing impossible tasks up to 73.9% of the time, revealing weak infeasibility detection capabilities
• →FeasiGen provides a reproducible methodology for constructing infeasible task benchmarks with over 94% accuracy
• →Multi-agent architectures significantly outperform single-agent systems at recognizing and stopping execution on infeasible tasks
• →Early detection of infeasible tasks could substantially reduce computational costs in resource-constrained deployments
• →Current evaluation frameworks fail to adequately measure agents' ability to recognize task impossibility and terminate appropriately