SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters

SAGA is a new distributed GPU scheduler that treats entire AI agent workflows as atomic units rather than individual inference calls, reducing task completion time by 1.64x compared to existing solutions. The system achieves this through workflow-aware scheduling, KV cache optimization, and fairness mechanisms, though with a tradeoff of 30% lower peak throughput suitable for latency-sensitive interactive deployments.

SAGA addresses a fundamental inefficiency in how current GPU schedulers handle compound AI workloads. Traditional request-level scheduling discards intermediate states between chained LLM calls, forcing agents to recompute context and inflating latency by 3-8x. This mismatch becomes critical as AI agent complexity grows—tasks requiring dozens of chained operations suffer compounding delays. SAGA's shift to program-level scheduling treats entire workflows as first-class units, enabling better resource management across correlated requests.

The technical innovation centers on three mechanisms: Agent Execution Graphs predict KV cache reuse patterns across tool boundaries, session-affinity batching co-locates related requests for faster context switching, and Agent Fair Share ensures equitable task-completion-time distribution. Testing on real multi-tenant GPU clusters demonstrates 1.64x latency reduction with 1.22x better memory utilization, while maintaining 99.2% SLO attainment under interference.

For the AI infrastructure market, SAGA's results highlight that latency optimization for interactive applications requires fundamentally rethinking scheduler abstractions. The 30% throughput reduction represents an intentional tradeoff—acceptance that compound AI serving prioritizes responsiveness over raw batch efficiency. This challenges the assumption that maximizing GPU utilization through traditional batching serves all workloads equally. Organizations deploying coding agents, browser automation, or multi-step reasoning systems face clear incentives to adopt workflow-aware schedulers, creating differentiation opportunities in the GPU cluster management space.

• →Workflow-level scheduling reduces compound AI task latency by 1.64x versus request-level scheduling with KV cache management.
• →SAGA maintains 1.22x better GPU memory utilization through intelligent cache reuse prediction across agent tool calls.
• →The scheduler sacrifices 30% peak throughput to optimize for latency, appropriate for interactive rather than batch-optimized deployments.
• →Session-affinity batching with work stealing balances co-location of correlated requests with global load distribution.
• →Results from 64-GPU clusters serving real agent workloads demonstrate practical feasibility with 99.2% SLO attainment under multi-tenant conditions.