RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment

Researchers introduce RE-PO (Robust Enhanced Policy Optimization), a new framework that addresses noise in human preference data used to train large language models. The method uses expectation-maximization to identify unreliable labels and reweight training data, improving alignment algorithm performance by up to 7% on benchmarks.

• →RE-PO addresses a critical problem in LLM training where human preference datasets contain substantial noise from annotator mistakes and inconsistent feedback.
• →The framework uses expectation-maximization to identify and reweight unreliable training labels, improving model alignment with human values.
• →RE-PO can be applied to enhance existing alignment methods including DPO, IPO, SimPO, and CPO algorithms.
• →Testing on Mistral and Llama 3 models showed up to 7% improvement in AlpacaEval 2 win rates compared to baseline methods.
• →The approach provides theoretical guarantees for recovering true noise levels in datasets under perfectly calibrated models.