Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
Researchers conducted interviews with 17 experienced developers to understand how they actually oversee autonomous software agents in practice, identifying four forms of oversight work (a priori control, co-planning, real-time monitoring, and post hoc review) and documenting practical challenges developers face when managing AI agents.
This empirical study bridges a critical gap between theoretical frameworks on AI agent oversight and real-world developer practices. As autonomous software agents become more prevalent in development workflows, understanding how practitioners actually manage and supervise these systems has become essential for building safer, more reliable AI tools.
The research reveals that developer oversight of agents is far more nuanced than existing literature suggests. Rather than being purely reactive—catching failures after they occur—developers employ preventative strategies including a priori control (setting constraints before execution) and co-planning (collaboratively designing agent behavior). This proactive approach contradicts assumptions in normative frameworks that treat oversight primarily as a retrospective activity. The study documents genuine challenges developers encounter, such as difficulty reviewing agent-generated code and determining when to trust agent outputs, alongside practical heuristics they've developed to address these issues, including relying on test results as proxies for code correctness.
These findings have significant implications for both AI safety and software engineering practice. For developers and teams deploying agent systems, the documented oversight challenges and emerging solutions provide actionable patterns for structuring their own agent management workflows. For AI system designers, the research highlights that oversight interfaces and agent behavior transparency require thoughtful design centered on human needs rather than theoretical best practices. The work suggests future tooling should support the full spectrum of oversight activities—not just detection mechanisms—while accounting for the limitations developers face when evaluating agent-generated artifacts.
- →Developers employ four distinct oversight strategies ranging from preventative controls to retrospective reviews, contradicting existing frameworks that emphasize reactive oversight
- →A priori control and proactive co-planning are common among experienced developers despite receiving limited attention in oversight research
- →Code review difficulty and output trustworthiness assessment emerge as primary oversight challenges without clear technical solutions
- →Developers rely on practical heuristics like test results as confidence measures, revealing potential gaps between theoretical oversight and pragmatic reality
- →Human-centered design of agent systems must support the full lifecycle of oversight work, not just post-failure analysis