Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization

Researchers propose MetaAPO, a new framework for aligning large language models with human preferences that dynamically balances online and offline training data. The method uses a meta-learner to evaluate when on-policy sampling is beneficial, resulting in better performance while reducing online annotation costs by 42%.

• →MetaAPO introduces a novel approach to preference optimization that dynamically couples data generation with model training.
• →The framework uses a lightweight meta-learner as an 'alignment gap estimator' to balance online and offline data quality.
• →Experiments show consistent outperformance across AlpacaEval 2, Arena-Hard, and MT-Bench benchmarks.
• →The method reduces online annotation costs by 42% compared to existing approaches.
• →The research addresses the critical challenge of distribution mismatch in LLM preference optimization.