Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Researchers propose a causally motivated method to reduce biases in reward models used for LLM alignment by identifying and suppressing neurons correlated with spurious features like response length. The technique achieves comparable performance to much larger models while editing less than 2% of neurons, suggesting biases are concentrated in early network layers.

This research addresses a fundamental problem in AI safety: reward models (RMs) that guide large language model training often rely on shortcuts rather than genuine preference signals. Response length bias exemplifies this issue—models learn to associate longer responses with higher rewards regardless of quality. The paper's innovation lies in its causal approach to identifying which neurons encode bias signals, then surgically suppressing them rather than retraining entire models.

The work builds on growing recognition that neural networks encode spurious correlations in predictable ways. Previous mitigation efforts typically focused narrowly on single bias types, creating trade-offs between reducing bias and maintaining performance. By targeting multiple bias dimensions simultaneously through neuron-level intervention, this method demonstrates that bias exploitation is a discrete, identifiable phenomenon rather than an entangled aspect of model behavior.

The practical implications are significant for AI developers. Small RMs (2B-7B parameters) with this debiasing technique match performance of substantially larger 70B models on standard benchmarks like AlpacaEval and MT-Bench. This efficiency gain could democratize high-quality preference annotation and model alignment for organizations with limited computational resources. The finding that bias signals concentrate in early layers provides a mechanistic understanding of how models exploit spurious features, potentially informing future training approaches.

Future work should explore whether this intervention method generalizes to other model architectures, training paradigms, and bias types beyond those tested. Understanding whether debiasing at inference time produces more robust models than training-time approaches remains an open question.

• →Causally-motivated neuron suppression reduces multiple bias types in reward models without performance degradation.
• →Small 2B-7B parameter RMs with debiasing match 70B model performance on major LLM benchmarks.
• →Bias signals concentrate primarily in early transformer layers, revealing internal mechanisms of shortcut exploitation.
• →Editing less than 2% of neurons achieves significant debiasing, suggesting highly localized bias encoding.
• →Method enables efficient preference annotation for model alignment across diverse bias dimensions.