From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents

A comprehensive survey examines evidence tracing and execution provenance in LLM agents—mechanisms for tracking how autonomous AI systems arrive at decisions by documenting retrieved evidence, tool interactions, and memory influences. This research addresses critical gaps in verifying, debugging, and auditing agent behavior beyond simple output accuracy, proposing frameworks and taxonomies for process-level accountability in AI systems.

The emergence of autonomous LLM agents capable of tool use, memory management, and multi-step reasoning has created a fundamental verification problem: understanding not just what an agent outputs, but how and why it reached that conclusion. This survey tackles that challenge by systematizing approaches to evidence tracing—connecting data sources, intermediate claims, and final answers through an execution path—and execution provenance, which documents the lineage of decisions throughout agent operation.

The research responds to genuine operational needs in production AI systems. As LLM agents handle increasingly complex tasks involving external APIs, knowledge retrieval, and persistent memory, failures become harder to diagnose and results harder to validate. A wrong answer might stem from a faulty tool call, corrupted memory state, poor retrieval, or reasoning error—problems invisible to traditional accuracy metrics. This transparency gap creates liability and trust issues, particularly in regulated domains like finance and healthcare where audit trails and explainability are non-negotiable.

The survey's taxonomy and methodological framework directly benefit AI engineers and researchers building enterprise-grade agents. By standardizing how provenance is represented, captured, and evaluated, the work enables better debugging tools, safety mechanisms, and recovery strategies. The shift toward process-level accountability rather than output-only evaluation signals maturation in how the industry assesses AI system reliability.

Looking forward, adoption of unified trace schemas and provenance-aware safety mechanisms will likely become table stakes for production AI systems, particularly as regulatory scrutiny increases. Organizations deploying autonomous agents will increasingly demand and expect detailed execution transparency, driving tooling development and standardization efforts across the AI infrastructure ecosystem.

• →Evidence tracing creates accountability by documenting how LLM agents connect data sources, tool outputs, and memory to final answers throughout execution.
• →Current evaluation metrics focusing on final-answer accuracy miss critical debugging information about tool-use justification, retrieval grounding, and failure origins.
• →Standardized provenance representation and execution trace schemas remain open challenges requiring industry consensus to enable interoperable auditing and safety mechanisms.
• →Production AI systems increasingly demand process-level transparency for regulatory compliance and liability management, not just output correctness.
• →Provenance-bearing memory and runtime guardrails represent emerging methodological directions for building trustworthy autonomous agent architectures.