AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

Researchers introduce AGENTSERVESIM, a hardware-aware simulator designed to evaluate serving policies for multi-turn LLM agents without requiring expensive accelerator deployments. The simulator accurately reproduces real-system performance within 6% error while running on standard CPUs, enabling scalable exploration of agent-serving policies across different hardware configurations and workload scenarios.

AGENTSERVESIM addresses a critical infrastructure challenge in modern AI systems: the complexity of serving stateful, multi-turn LLM agents that interleave model inference with external tool calls. Traditional LLM serving simulators were built for stateless request-processing workloads and fail to capture the unique dynamics of agent execution, including turn dependencies, KV-cache residency during tool-induced delays, and cross-turn cache locality patterns. This gap has forced researchers and practitioners to conduct expensive trial-and-error deployments on GPU/TPU clusters to evaluate serving policies.

The simulator's architecture reflects the actual operational challenges of agent serving. Its Program Orchestrator maintains execution order and program identity, while the Tool Simulator models realistic gaps when agents call external APIs. The Session-Aware Router implements cache-aware dispatch strategies, and the KV Residency Model tracks memory placement decisions across HBM, host DRAM, and CXL hierarchies—key considerations as memory becomes a bottleneck in large-scale inference deployments.

For the AI infrastructure industry, AGENTSERVESIM democratizes serving optimization research. Currently, only well-resourced organizations can afford the compute costs to experiment with scheduling and caching policies. By enabling commodity CPU-based simulation with 6% accuracy, the tool reduces barriers to innovation in serving systems. This extends research accessibility beyond major cloud providers and large labs, potentially accelerating improvements in inference efficiency.

The work signals growing maturity in agent-based AI applications, transitioning from theoretical capability discussions to operational deployment challenges. As multi-turn agents become production workloads, infrastructure optimization becomes economically critical. Future development should focus on expanding simulator scope to multi-instance scheduling, heterogeneous hardware setups, and real-world tool latency distributions.

• →AGENTSERVESIM enables accurate simulation of multi-turn LLM agent serving on commodity CPUs with only 6% error compared to real hardware deployments.
• →The simulator addresses unique dynamics of stateful agent execution including turn dependencies, tool-induced gaps, and cross-turn KV-cache locality that prior serving simulators ignored.
• →Hardware-aware KV residency modeling across HBM, DRAM, and CXL hierarchies provides realistic memory constraint evaluation for inference optimization.
• →Commodity CPU-based simulation democratizes serving policy research by reducing dependency on expensive accelerator time for infrastructure experimentation.
• →The work reflects industry maturation of multi-turn agents from theoretical capability to operational deployment requiring infrastructure-level optimization.