Characterizing Software Aging in GPU-Based LLM Serving Systems

Researchers conducted a 216-hour empirical study on software aging in GPU-based LLM serving systems, revealing statistically significant memory leaks across deployments. The findings highlight that memory degradation rates vary substantially based on serving runtime and configuration, establishing a reproducible framework for studying aging patterns in systems combining Python hosts and CUDA devices.

Software aging—the gradual performance degradation of long-running systems—has traditionally been studied in CPU-centric environments with predictable workloads. This research extends that methodology into LLM serving infrastructure, where conditions are markedly different: variable request costs spanning multiple orders of magnitude, heterogeneous computing across host and GPU device, and rapidly evolving software dependencies. The 216-hour experimental campaign under controlled stress conditions provides empirical evidence that memory aging is not merely theoretical but measurably significant in production-relevant scenarios.

The study's primary contribution lies in demonstrating that aging characteristics vary dramatically with deployment choices. This suggests that operators cannot apply generic rejuvenation strategies across LLM serving systems; they must profile their specific runtime and configuration combinations to understand leak rates and degradation patterns. The dependence on serving runtime implies that framework choices—whether vLLM, Ray Serve, or others—materially affect system reliability over time.

For infrastructure operators and LLM service providers, these findings carry immediate practical implications. Memory leaks compound over extended deployment periods, forcing periodic restarts that disrupt service availability and increase operational costs. The reproducible framework enables teams to characterize their own aging profiles and plan maintenance windows accordingly. For the broader AI systems community, this work bridges software reliability engineering and LLM operations, establishing baselines for future rejuvenation strategies and architectural improvements.

• →Memory aging in GPU LLM serving systems shows statistically significant degradation tied to specific serving runtimes and configurations.
• →Memory leak rates vary by orders of magnitude depending on deployment choices, requiring environment-specific profiling rather than generic solutions.
• →Extended LLM service deployments accumulate memory degradation over hours to days, necessitating planned rejuvenation cycles.
• →The study provides a reproducible framework applicable across heterogeneous host-device architectures with variable workload costs.
• →LLM serving infrastructure reliability improvements require addressing software aging at both Python host and CUDA device layers simultaneously.