The Token Not Taken: Sampling, State, and the Variability of AI Agent Outputs

A new arXiv paper analyzes the sources of variability in agentic AI systems, distinguishing between token-sampling randomness intrinsic to foundation models and external factors like environmental changes and infrastructure effects. The research clarifies when AI agent outputs are genuinely stochastic versus reproducible, with implications for understanding AI reliability in production deployments.

This paper addresses a critical gap in how the AI community conceptualizes non-determinism in agentic systems. Current deployed AI agents frequently produce different outputs for identical inputs, but the field has lacked a rigorous framework for understanding why. The manuscript separates intrinsic sources—primarily token sampling during language model inference—from extrinsic sources including live data feeds, environmental state changes, and infrastructure variations. This distinction matters because it determines which variability is inherent to the AI model itself versus which stems from operational contexts.

The research builds on growing recognition that foundation models, when embedded in orchestration loops with tool calls and state management, amplify small stochastic decisions into divergent execution paths. A single sampled token difference can cascade into different tool selections, code paths, or search queries, fundamentally altering agent behavior. Understanding these layers enables developers to reason about reproducibility and reliability—critical for applications requiring consistency.

For the AI industry, this work has immediate practical value. Teams deploying production agents need to distinguish controllable variance from unavoidable randomness to set appropriate reliability expectations. The framework enables better testing practices and clearer communication with stakeholders about when deterministic inputs should guarantee identical outputs. In high-stakes applications like code generation, financial analysis, or system automation, this clarity between sampling noise and environmental factors directly impacts risk assessment and debugging strategies, potentially reducing costly failures from misattributed variability.

• →Agentic AI variability stems from multiple distinct layers: token sampling (intrinsic), environment changes, and infrastructure effects (extrinsic).
• →Small differences in sampled tokens can propagate into completely different tool calls, code paths, and agent decisions.
• →Deterministic model execution does not guarantee identical deployed behavior due to external factors beyond the foundation model.
• →Separating intrinsic and extrinsic variability sources enables better reproducibility testing and reliability engineering for AI agents.
• →Understanding these layers clarifies when agent behavior is genuinely stochastic versus when it's reproducible under matched conditions.