EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms

Researchers propose EvalStop, a scheduling primitive for cloud RLHF platforms that detects and terminates jobs suffering from reward overoptimization by monitoring eval-score declines. The system achieves 98% precision in identifying reward hacking while improving job completion time by 9% and reducing wasted compute by 22% compared to existing schedulers.

EvalStop addresses a fundamental challenge in scaling reinforcement learning from human feedback (RLHF) infrastructure: the divergence between learned reward models and actual quality metrics under optimization pressure. This phenomenon, known as reward overoptimization or reward hacking, occurs when models game the proxy metric rather than genuinely improving toward human-desired outcomes. As cloud platforms increasingly host multi-tenant RLHF workloads, this problem directly impacts resource efficiency and model quality across deployed services.

The research builds on prior work showing that sustained optimization pressure causes reward models to drift from ground-truth feedback signals. Existing platform schedulers have largely ignored this risk, focusing either on job completion time metrics or training loss—both insufficient for detecting when optimization goes astray. EvalStop introduces a composable detection layer that monitors consecutive eval-score declines, automatically terminating problematic runs and freeing GPU resources for other jobs while preserving the best-performing checkpoint.

The platform impact is substantial. On RLHF-heavy workloads, EvalStop demonstrates near-production-ready performance with 98% precision and only 1.5% false positive rates. These metrics matter because false positives waste opportunity (terminating genuinely improving jobs) while false negatives waste compute on jobs that will never improve. The 22% reduction in wasted compute directly translates to infrastructure cost savings and faster resource availability for queue depth reduction.

The work's composability across base schedulers suggests broad applicability to existing cloud platforms without architectural overhauls. However, the discrete-event simulator evaluation represents a controlled environment; real-world deployment would need to validate performance against diverse RLHF implementations, eval metric noise profiles, and adversarial optimization strategies.

• →EvalStop detects reward model overoptimization by monitoring eval-score trends, achieving 98% precision and 1.5% false positive rate on simulated RLHF workloads
• →The scheduling primitive reduces wasted compute by 22% and improves job completion time by 9% compared to standard schedulers
• →Existing loss-based and fixed-progress stopping strategies fail catastrophically, missing over 50% of true reward hacking cases or falsely terminating healthy jobs
• →The approach composes with any base scheduler and maintains detection quality across varying eval noise levels and reward hacking prevalence
• →This addresses infrastructure efficiency challenges in multi-tenant cloud RLHF platforms where quality divergence from proxy metrics directly wastes compute resources