The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation

Research reveals that RLHF-aligned language models suffer from 'alignment tax' - producing homogenized responses that severely impair uncertainty estimation methods. The study found 40-79% of questions on TruthfulQA generate nearly identical responses, with alignment processes like DPO being the primary cause of this response homogenization.

• →RLHF-aligned models show severe response homogenization, with 40-79% of questions producing single semantic clusters across samples.
• →Traditional sampling-based uncertainty methods become ineffective (AUROC=0.500) on homogenized responses, though token entropy retains some signal.
• →The alignment tax is primarily caused by DPO training rather than supervised fine-tuning, as confirmed through training stage ablations.
• →The severity of response homogenization varies by model family and scale, affecting uncertainty estimation across multiple benchmarks.
• →A proposed cascade method using orthogonal uncertainty signals can improve selective prediction accuracy from 84.4% to 93.2% while reducing costs by 57%.