Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation

Researchers introduced GAIATrace, a token-level trace dataset documenting how state-of-the-art agentic AI systems (MiroThinker and OWL) execute general tasks, alongside Vidur-Agent, a simulator enabling reproducible system evaluation. This work addresses the black-box nature of agentic AI by providing unprecedented visibility into reasoning processes and system-level behavior.

Understanding agentic AI systems has proven challenging due to their non-deterministic execution paths, high evaluation costs, and reliance on proprietary models. GAIATrace fundamentally shifts this landscape by capturing token-level traces across multiple state-of-the-art agentic systems executing diverse general-purpose tasks. Unlike previous trace datasets, this resource preserves full reasoning tokens and task-level structures, enabling researchers to examine not just outputs but the complete decision-making processes underlying agent behavior.

The release of this dataset reflects a maturation in AI systems research. As agentic systems become increasingly complex with iterative planning and tool use, the industry requires better mechanisms to understand their behavior and failure modes. This trace dataset addresses a critical gap: most agentic system evaluations occur within proprietary environments or rely on limited sampling, obscuring systemic patterns that emerge across diverse task types.

Vidur-Agent, the accompanying simulator, extends the practical utility of GAIATrace by enabling low-cost, reproducible experiments. Developers can now test architectural modifications and design choices without incurring the computational costs of executing full agentic systems. This democratizes agentic systems research and accelerates optimization cycles.

For the AI development community, GAIATrace establishes a foundation for comparative systems analysis. Researchers can now identify which design choices yield superior performance characteristics, understand failure patterns across task categories, and design more efficient agentic architectures. The findings about how system design shapes agent behavior provide actionable insights for future development, potentially improving both performance and resource efficiency in production agentic systems.

• →GAIATrace provides the first comprehensive token-level trace dataset for agentic AI systems, capturing previously hidden reasoning processes and decision-making patterns.
• →Vidur-Agent simulator enables reproducible evaluation of agentic systems at a fraction of typical computational costs.
• →The dataset reveals how different architectural choices influence agentic system behavior on heterogeneous general-purpose tasks.
• →This research addresses critical visibility gaps in understanding non-deterministic AI systems and their failure modes.
• →The work establishes foundational tools for comparative systems analysis in agentic AI development.