AIBullisharXiv – CS AI · Jun 257/10
🧠Researchers demonstrate that reinforcement learning post-training for large language models can generate effective step-level reward signals without dedicated reward model training. The 'progress advantage' metric—derived from log-probability ratios between trained and reference policies—eliminates annotation overhead while matching or exceeding performance of purpose-built reward models across multiple applications.
AIBullisharXiv – CS AI · Jun 107/10
🧠Researchers identify a critical bias in Bradley-Terry loss, the standard objective for training reward models in LLM alignment, where gradient magnitudes are distorted by representation distance rather than prediction error. They propose NormBT, a lightweight normalization scheme that refocuses learning on actual ranking mistakes, demonstrating 5%+ improvements on fine-grained reasoning benchmarks.
AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers present three techniques for inference-time scaling that extend beyond verifiable domains by using intrinsic statistical signals from parallel samples to assess solution quality without ground truth. The methods—Intrinsic Selection, Intrinsic Particle Filtering, and Particle Distillation—improve performance on open-ended tasks like engineering design and clinical reasoning by 6-26% without requiring trained reward models.
AIBearisharXiv – CS AI · Jun 87/10
🧠Researchers reveal a significant gap between synthetic and real-world performance in LLM personalization systems by analyzing 550 human conversations across three stages: attribute extraction, attribute selection, and response generation. The study finds that current models struggle with human-aligned personalization and that learned reward models fail to adequately capture human preferences, highlighting fundamental limitations in how AI systems understand and incorporate user information.
AIBearisharXiv – CS AI · Jun 47/10
🧠Researchers demonstrate that offline bandit algorithms—used to evaluate machine learning models like image generators and LLMs—are vulnerable to adversarial attacks on their reward models. The study reveals that in high-dimensional settings, attackers can achieve near-perfect success rates with imperceptibly small perturbations to publicly available reward model weights, creating a critical security gap in AI evaluation systems.
🏢 Hugging Face
AINeutralarXiv – CS AI · Jun 27/10
🧠Researchers establish a theoretical framework explaining why large language models optimized through outcome-based reinforcement learning develop brittle reasoning despite strong benchmark performance. The study introduces 'Reward-Induced Manifold Collapse' and demonstrates that process reward models can prevent this failure mode by enforcing information constraints on reasoning steps.
AIBearisharXiv – CS AI · May 287/10
🧠Researchers demonstrate that single-axis bias mitigations in AI reward models often redirect optimization pressure to correlated biases rather than eliminating it—a failure mode called reward bias substitution. The study proves that successful mitigation, bias substitution, and overcorrection produce identical observable results under standard audit metrics, meaning current evaluation methods cannot distinguish between genuine fixes and problematic redirections.
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce Video Understanding Reward Bench (VURB), a comprehensive benchmark with 2,100 preference pairs for evaluating video reward models, alongside VUP-35K, a large-scale dataset of 35,000 preference examples. Two new models, VideoDRM and VideoGRM, achieve state-of-the-art performance on video understanding tasks, advancing multimodal AI capabilities beyond text and images.
AINeutralarXiv – CS AI · May 97/10
🧠Researchers introduce Chameleon, a dataset of 5,001 contextual psychological profiles revealing that 74% of user behavior variance stems from situational context (state) rather than personality traits (26%). The study finds language models are state-blind, responding similarly regardless of context, while reward models inconsistently evaluate the same users differently across scenarios.
AIBullisharXiv – CS AI · May 97/10
🧠Researchers introduce SelectiveRM, an optimal transport-based framework that improves reward model training for large language models by handling noisy preference data. The approach uses joint consistency discrepancy and partial transport mechanisms to automatically filter out contradictory samples, theoretically optimizing cleaner risk bounds and outperforming existing methods.
AIBearisharXiv – CS AI · May 77/10
🧠Researchers found that reward models used to align large language models often fail to capture socially desirable preferences, preferring biased, unsafe, or unethical responses across domains like bias, safety, and morality. The study reveals a critical misalignment between how reward models are currently evaluated and their actual performance on social intelligence tasks, exposing a fundamental gap in LLM safety infrastructure.
AIBullisharXiv – CS AI · May 17/10
🧠Researchers propose a causally motivated method to reduce biases in reward models used for LLM alignment by identifying and suppressing neurons correlated with spurious features like response length. The technique achieves comparable performance to much larger models while editing less than 2% of neurons, suggesting biases are concentrated in early network layers.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers identified persistent biases in high-quality language model reward systems, including length bias, sycophancy, and newly discovered model-style and answer-order biases. They developed a mechanistic reward shaping method to reduce these biases without degrading overall reward quality using minimal labeled data.
AIBullisharXiv – CS AI · Mar 47/103
🧠Researchers introduce Skywork-Reward-V2, a suite of AI reward models trained on SynPref-40M, a massive 40-million preference pair dataset created through human-AI collaboration. The models achieve state-of-the-art performance across seven major benchmarks by combining human annotation quality with AI scalability for better preference learning.
AINeutralarXiv – CS AI · Mar 37/103
🧠A comprehensive study of 10 leading reward models reveals they inherit significant value biases from their base language models, with Llama-based models preferring 'agency' values while Gemma-based models favor 'communion' values. This bias persists even when using identical preference data and training processes, suggesting that the choice of base model fundamentally shapes AI alignment outcomes.
AINeutralarXiv – CS AI · Feb 277/107
🧠Researchers propose a new approach for training AI models to generate correct answers from demonstrations, using imitation learning in contextual bandits rather than traditional supervised fine-tuning. The method achieves better sample complexity and works with weaker assumptions about the underlying reward model compared to existing likelihood-maximization approaches.
AIBullisharXiv – CS AI · Feb 277/106
🧠Researchers introduce Dual-Iterative Preference Optimization (Dual-IPO), a new method that iteratively improves both reward models and video generation models to create higher-quality AI-generated videos better aligned with human preferences. The approach enables smaller 2B parameter models to outperform larger 5B models without requiring manual preference annotations.
AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers propose FiMi-RM, a framework that identifies and corrects length bias in reward models used for RLHF training of large language models. The approach uses a lightweight fitting model to capture non-linear length-reward relationships and decouples them from preference scoring, reducing AI systems' tendency to favor longer responses regardless of quality.
AINeutralarXiv – CS AI · Jun 196/10
🧠Researchers introduce AURA, a framework that improves the reliability of using large language models as judges for evaluating generated text by iteratively learning human-consistency patterns and prioritizing uncertain comparisons for human review. The approach addresses the core challenge that LLM judges often reflect their own biases rather than genuine human preferences, even when some human feedback is available.
AIBullisharXiv – CS AI · Jun 96/10
🧠Researchers introduce AdaGRPO, a reinforcement learning framework that selectively applies reward signals in generative recommendation systems rather than uniformly, addressing the problem of noisy reward models trained on biased data. The approach combines supervised learning with adaptive gating mechanisms and demonstrates significant improvements in e-commerce recommendation metrics and production performance.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers propose using distributional reward models instead of scalar models to address reward hacking in RLHF, where AI policies exploit errors in reward models. A unified mathematical framework shows that pessimistic reward adjustment through KL regularization recovers existing ensemble aggregation methods as special cases, providing theoretical clarity on uncertainty handling in AI alignment.
AINeutralarXiv – CS AI · Jun 86/10
🧠Researchers introduce StainFlow, a process reward model that improves reinforcement learning for GUI agents by tracking entity states and dynamically linking evidence across trajectories. The method achieves 3.2% relative improvement in online RL success and 1.8% improvement in trajectory completion accuracy on benchmark tasks.
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers propose AISP (Adaptive Importance Sampling on Pre-logits), a test-time alignment method for large language models that uses Gaussian perturbations to optimize reward signals without expensive fine-tuning. The technique outperforms existing sampling-based approaches and represents progress in making LLM alignment more computationally efficient.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce Demo2Reward, a test-time optimization technique that improves Vision-Language Model (VLM) reward models by refining prompts based on a small number of expert demonstrations. The method reduces false positives in reward prediction without requiring additional model training, enabling more effective reinforcement learning in robotics applications including real-world scenarios.
AINeutralarXiv – CS AI · Jun 16/10
🧠Researchers formalize calibration concepts for probabilistic label ranking, revealing that popular models often fail to align predicted probabilities with actual outcome frequencies. The framework uncovers a gap between sub-ranking and top-k calibration metrics, with implications for RLHF reward models used in AI systems.
