Human-Less LLM Serving: Quantifying the Human Tax on Throughput

Researchers quantify a significant efficiency cost in LLM serving systems: meeting latency targets (TTFT and TPOT) designed for human users reduces throughput by 60-93% for AI workloads that don't require human-perceptible latency. The study demonstrates that one-size-fits-all SLA configurations waste substantial computational resources when applied to programmatic AI-to-AI tasks.

Current LLM serving infrastructure prioritizes metrics like Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT) because these directly affect user experience for human interactions. However, an emerging class of workloads—where AI systems programmatically call LLMs in loops without human observation—doesn't benefit from these optimizations yet bears their computational cost.

This research bridges a gap between infrastructure design and evolving use cases. As AI agents, autonomous systems, and multi-step reasoning tasks become more prevalent, the efficiency losses compound. The 60-93% throughput penalty scales dramatically with context length, reaching critical levels at 64K tokens. This suggests that current serving systems are fundamentally misaligned with next-generation workload patterns.

The implications extend across the AI infrastructure stack. Cloud providers, LLM API vendors, and organizations deploying reasoning-heavy systems face unnecessary operational costs. At scale, this inefficiency translates to higher inference expenses and wasted computational capacity. Companies like Anthropic, OpenAI, and emerging inference-optimization startups could capture significant value by offering workload-aware SLA configurations.

The prototype demonstration of "human-less serving" validates that practical solutions exist. The industry appears poised to transition from monolithic serving architectures to differentiated configurations. This evolution mirrors previous infrastructure maturation cycles where general-purpose systems eventually fragment into specialized tiers. Organizations running batch reasoning workloads should monitor whether their inference providers offer explicit optimization for human-less patterns.

• →LLM serving systems sacrifice 60-93% throughput to meet human-focused latency SLOs on workloads that don't require them
• →The efficiency penalty grows substantially with context length, becoming critical above 64K tokens
• →AI-to-AI programmatic tasks represent an emerging workload class with fundamentally different performance requirements than interactive systems
• →Serving infrastructure requires workload-class-aware configurations rather than uniform SLA application across all traffic
• →Human-less serving prototypes demonstrate practical feasibility for optimizing throughput in non-human-observable scenarios