AIBearisharXiv – CS AI · 3d ago7/10
🧠Researchers demonstrate that single-axis bias mitigations in AI reward models often redirect optimization pressure to correlated biases rather than eliminating it—a failure mode called reward bias substitution. The study proves that successful mitigation, bias substitution, and overcorrection produce identical observable results under standard audit metrics, meaning current evaluation methods cannot distinguish between genuine fixes and problematic redirections.
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce Video Understanding Reward Bench (VURB), a comprehensive benchmark with 2,100 preference pairs for evaluating video reward models, alongside VUP-35K, a large-scale dataset of 35,000 preference examples. Two new models, VideoDRM and VideoGRM, achieve state-of-the-art performance on video understanding tasks, advancing multimodal AI capabilities beyond text and images.
AIBullisharXiv – CS AI · May 97/10
🧠Researchers introduce SelectiveRM, an optimal transport-based framework that improves reward model training for large language models by handling noisy preference data. The approach uses joint consistency discrepancy and partial transport mechanisms to automatically filter out contradictory samples, theoretically optimizing cleaner risk bounds and outperforming existing methods.
AINeutralarXiv – CS AI · May 97/10
🧠Researchers introduce Chameleon, a dataset of 5,001 contextual psychological profiles revealing that 74% of user behavior variance stems from situational context (state) rather than personality traits (26%). The study finds language models are state-blind, responding similarly regardless of context, while reward models inconsistently evaluate the same users differently across scenarios.
AIBearisharXiv – CS AI · May 77/10
🧠Researchers found that reward models used to align large language models often fail to capture socially desirable preferences, preferring biased, unsafe, or unethical responses across domains like bias, safety, and morality. The study reveals a critical misalignment between how reward models are currently evaluated and their actual performance on social intelligence tasks, exposing a fundamental gap in LLM safety infrastructure.
AIBullisharXiv – CS AI · May 17/10
🧠Researchers propose a causally motivated method to reduce biases in reward models used for LLM alignment by identifying and suppressing neurons correlated with spurious features like response length. The technique achieves comparable performance to much larger models while editing less than 2% of neurons, suggesting biases are concentrated in early network layers.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers identified persistent biases in high-quality language model reward systems, including length bias, sycophancy, and newly discovered model-style and answer-order biases. They developed a mechanistic reward shaping method to reduce these biases without degrading overall reward quality using minimal labeled data.
AIBullisharXiv – CS AI · Mar 47/103
🧠Researchers introduce Skywork-Reward-V2, a suite of AI reward models trained on SynPref-40M, a massive 40-million preference pair dataset created through human-AI collaboration. The models achieve state-of-the-art performance across seven major benchmarks by combining human annotation quality with AI scalability for better preference learning.
AINeutralarXiv – CS AI · Mar 37/103
🧠A comprehensive study of 10 leading reward models reveals they inherit significant value biases from their base language models, with Llama-based models preferring 'agency' values while Gemma-based models favor 'communion' values. This bias persists even when using identical preference data and training processes, suggesting that the choice of base model fundamentally shapes AI alignment outcomes.
AIBullisharXiv – CS AI · Feb 277/106
🧠Researchers introduce Dual-Iterative Preference Optimization (Dual-IPO), a new method that iteratively improves both reward models and video generation models to create higher-quality AI-generated videos better aligned with human preferences. The approach enables smaller 2B parameter models to outperform larger 5B models without requiring manual preference annotations.
AINeutralarXiv – CS AI · Feb 277/107
🧠Researchers propose a new approach for training AI models to generate correct answers from demonstrations, using imitation learning in contextual bandits rather than traditional supervised fine-tuning. The method achieves better sample complexity and works with weaker assumptions about the underlying reward model compared to existing likelihood-maximization approaches.
AIBullisharXiv – CS AI · 4d ago6/10
🧠Researchers demonstrate that cross-lingual contrastive preference tuning (CroCo) enables large language models to improve performance across 14 languages without language-specific annotations by leveraging English-trained reward models. The method shows consistent gains in both structured and open-ended generation tasks across multiple languages while avoiding catastrophic forgetting.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers introduce FormalRewardBench, the first benchmark for evaluating reward models in formal theorem proving using Lean 4. The benchmark reveals that frontier LLMs like Claude Opus outperform specialized theorem provers at evaluating proof quality, suggesting that theorem proving ability does not transfer to proof evaluation tasks.
🧠 Claude🧠 Opus
AINeutralarXiv – CS AI · May 116/10
🧠Researchers propose using conditional optimal transport to improve calibration of Process Reward Models (PRMs) used in AI inference-time scaling, addressing the problem of overestimated success probabilities. The method enables better confidence bounds for mathematical reasoning tasks and improves downstream performance in Best-of-N selection frameworks.
AINeutralarXiv – CS AI · May 76/10
🧠Researchers introduce StoryRMB, the first benchmark for evaluating reward models on story generation preferences, and develop StoryReward, a specialized reward model achieving 66.3% accuracy where existing models struggle. The work addresses the challenge of modeling subjective human preferences in narrative generation, enabling better alignment between LLM-generated stories and human expectations.
AIBullisharXiv – CS AI · Mar 176/10
🧠Researchers have developed EvolvR, a self-evolving framework that improves AI's ability to evaluate and generate stories through pairwise reasoning and multi-agent data filtering. The system achieves state-of-the-art performance on three evaluation benchmarks and significantly enhances story generation quality when used as a reward model.
AIBullisharXiv – CS AI · Mar 166/10
🧠Researchers propose AdaBoN, an adaptive Best-of-N alignment method that improves computational efficiency in language model alignment by allocating inference-time compute based on prompt difficulty. The two-stage algorithm outperforms uniform allocation strategies while using 20% less computational budget.
AINeutralarXiv – CS AI · Mar 36/1012
🧠RubricBench is a new benchmark with 1,147 pairwise comparisons designed to evaluate rubric-based assessment methods for Large Language Models. Research reveals a significant gap between human-annotated and AI-generated rubrics, showing that current state-of-the-art models struggle to autonomously create valid evaluation criteria.
AINeutralarXiv – CS AI · Mar 26/1010
🧠Researchers introduce RewardUQ, a unified framework for evaluating uncertainty quantification in reward models used to align large language models with human preferences. The study finds that model size and initialization have the most significant impact on performance, while providing an open-source Python package to advance the field.
AIBullisharXiv – CS AI · Mar 27/1015
🧠Researchers introduce R2M (Real-Time Aligned Reward Model), a new framework for Reinforcement Learning from Human Feedback (RLHF) that addresses reward overoptimization in large language models. The system uses real-time policy feedback to better align reward models with evolving policy distributions during training.
AINeutralarXiv – CS AI · Mar 27/1015
🧠Research reveals that reward model accuracy alone doesn't determine effectiveness in RLHF systems. The study proves that low reward variance can create flat optimization landscapes, making even perfectly accurate reward models inefficient teachers that underperform less accurate models with higher variance.
AIBullishSynced Review · Apr 116/106
🧠DeepSeek AI has published research detailing a new technique called SPCT for enhancing the scalability of general reward models during inference. The development signals progress toward their next-generation R2 model with improved inference scaling capabilities.
AINeutralarXiv – CS AI · Mar 95/10
🧠Researchers revisited Best-of-N (BoN) sampling for AI alignment and found it's actually optimal when evaluated using win-rate metrics rather than expected true reward. They propose a variant that eliminates reward-hacking vulnerabilities while maintaining optimal performance.
AINeutralarXiv – CS AI · Mar 34/106
🧠Researchers introduce CMI-RewardBench, a comprehensive evaluation framework for music generation AI models that can process multimodal inputs including text, lyrics, and audio. The system includes a 110k sample preference dataset and reward models that show strong correlation with human judgments for music quality assessment.
AINeutralOpenAI News · Oct 191/107
🧠The article appears to discuss scaling laws related to reward model overoptimization in AI systems. However, the article body is empty, making it impossible to provide meaningful analysis of the content or implications.
