BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Researchers introduce BiasGRPO, a novel framework using Group Relative Policy Optimization to mitigate social bias in Large Language Models more effectively than existing methods. The approach stabilizes training in high-variance reward landscapes by normalizing rewards across sampled completions, outperforming Direct Preference Optimization and Proximal Policy Optimization while maintaining computational efficiency.

BiasGRPO addresses a fundamental challenge in LLM alignment: social bias mitigation lacks objective ground truth, making it inherently subjective and high-variance compared to verifiable tasks. Existing approaches face significant limitations—DPO's offline training restricts exploration capabilities, while PPO's reliance on critic estimates creates training instability. The proposed framework substitutes traditional value functions with group-relative baselines, normalizing rewards across completion cohorts to reduce variance while preserving online training's exploration benefits.

This research emerges amid growing industry focus on responsible AI deployment. As LLMs become more prevalent in production systems, bias mitigation has shifted from academic interest to practical necessity. Multiple organizations face reputational and regulatory pressures to demonstrate bias controls, creating demand for reliable mitigation techniques. The creation of a compute-efficient, custom bias reward model addresses deployment constraints that practitioners face when integrating alignment techniques into existing systems.

The framework's superior performance across benchmarks suggests meaningful improvements for developers building production LLM applications. Organizations implementing multi-objective RLHF pipelines can now access a more stable training approach without prohibitive computational overhead. The released bias reward model provides immediate utility for teams seeking to enhance fairness without extensive custom development. However, practical adoption depends on integration compatibility with existing training infrastructures and validation across diverse real-world bias scenarios beyond the tested benchmarks.

• →BiasGRPO stabilizes bias mitigation by normalizing rewards across grouped completions rather than relying on potentially unreliable critic estimates.
• →The framework outperforms DPO and PPO on multiple benchmarks while maintaining computational efficiency suitable for production deployment.
• →A custom, compute-efficient bias reward model is released as a reusable resource for multi-objective RLHF pipelines.
• →Group-relative baselines enable exploration benefits of online training while reducing high-variance reward landscape instability.
• →The approach addresses a critical industry need for reliable bias mitigation techniques without knowledge degradation.