50 articles tagged with #preference-learning. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose 'Markov decision contests' as a new reinforcement learning framework that leverages pairwise preferences instead of scalar rewards, proving that stationary Markov policies are optimal and demonstrating superior learning efficiency in long-horizon problems compared to existing methods.
Researchers propose SelSkill, a machine learning framework that improves how AI agents decide whether to invoke specific skills during task execution. The method demonstrates significant performance improvements on benchmark tasks by learning when to use skills versus skip them, addressing a gap in existing agentic AI systems that struggle with unnecessary skill invocations.
Researchers introduce Reward Partition Optimization (RPO), a new method for training language models that eliminates the need for value function estimation in preference-based learning. RPO simplifies the optimization process by normalizing rewards through partition-based formulations, demonstrating superior performance compared to existing approaches like DRO and KTO across multiple model architectures.
Researchers propose Bayesian Non-Negative Reward Model (BNRM), a framework that addresses reward hacking vulnerabilities in reinforcement learning from human feedback (RLHF) systems used to align large language models. The approach combines non-negative factor analysis with preference modeling to create more robust, interpretable reward systems resistant to biases and distribution shifts.
Researchers propose FedVPA-GP, a federated learning framework that enables privacy-preserving alignment of large language models while preserving diverse user preferences instead of averaging them into a single monolithic reward model. The approach uses a Gumbel-Softmax prior and orthogonal loss to prevent posterior collapse and successfully disentangles conflicting user intents in decentralized settings.
A new arXiv survey reframes large language model alignment tuning through a data-centric lens, decomposing alignment data construction into three stages: response synthesis, preference evaluation, and preference instantiation. By organizing existing alignment methods into a unified taxonomy, the research identifies design trade-offs and failure modes while establishing principles for improving alignment data pipeline design.
Researchers propose novel algorithms (LDB-DF and NDB-DF) for contextual dueling bandits that handle delayed feedback—a critical real-world constraint in recommender systems and LLM alignment. The breakthrough involves an Inverse Probability Weighting mechanism that eliminates bias from delayed observations, achieving theoretical regret bounds of O(d√T) for linear settings.
Researchers propose a new approach to embedding text for collective decision-making that prioritizes preferential similarity over semantic similarity. The method uses synthetic training data to separate preference signals (stance and values) from semantic nuisance (style and wording), improving preference prediction across deliberation datasets.
Researchers present a theoretical framework for inferring the preferences and reward functions of learning agents through observation, extending inverse reinforcement learning beyond its traditional assumption that observed agents act optimally. The work establishes mathematical guarantees for preference learning algorithms when agents are either no-regret learners or converge to optimal Boltzmann policies.
Researchers introduce MOCI (Multi-Objective Constraint Inference), a novel framework that uses inverse reinforcement learning to extract safety constraints and individual preferences from diverse expert demonstrations where multiple experts have different objectives. The approach addresses limitations in existing methods that assume homogeneous expert behavior and offers improved computational efficiency.
Researchers introduce DT-PBO, a tree-based surrogate model for Preferential Bayesian Optimization that prioritizes interpretability over traditional Gaussian Process approaches. The method achieves competitive performance on benchmark functions while providing transparent insights into decision-maker preferences, addressing critical needs in high-stakes domains like healthcare.
Researchers introduce StoryRMB, the first benchmark for evaluating reward models on story generation preferences, and develop StoryReward, a specialized reward model achieving 66.3% accuracy where existing models struggle. The work addresses the challenge of modeling subjective human preferences in narrative generation, enabling better alignment between LLM-generated stories and human expectations.
Researchers propose a framework that treats clinician overrides of AI recommendations as preference signals for training clinical decision-support systems in value-based care settings. The approach combines preference learning with capability modeling to improve AI alignment with patient outcomes rather than encounter economics, addressing a failure mode called suppression bias.
A new arXiv paper argues that AI alignment cannot rely solely on stated principles because their real-world application requires contextual judgment and interpretation. The research shows that a significant portion of preference-labeling data involves principle conflicts or indifference, meaning principles alone cannot determine decisions—and these interpretive choices often emerge only during model deployment rather than in training data.
Researchers demonstrate that looped transformers like Ouro-2.6B encode human preferences relationally rather than independently, with pairwise evaluators achieving 95.2% accuracy compared to 21.75% for independent classification. The study reveals that preference encoding is fundamentally relational, functioning as an internal consistency probe rather than a direct predictor of human annotations.
Researchers propose using Inductive Learning of Answer Set Programs (ILASP) to create interpretable approximations of neural networks trained on preference learning tasks. The approach combines dimensionality reduction through Principal Component Analysis with logic-based explanations, addressing the challenge of explaining black-box AI models while maintaining computational efficiency.
Research from arXiv shows that Active Preference Learning (APL) provides minimal improvements over random sampling in training modern LLMs through Direct Preference Optimization. The study found that random sampling performs nearly as well as sophisticated active selection methods while being computationally cheaper and avoiding capability degradation.
Researchers propose Preference-based Constrained Reinforcement Learning (PbCRL), a new approach for safe AI decision-making that learns safety constraints from human preferences rather than requiring extensive expert demonstrations. The method addresses limitations in existing Bradley-Terry models by introducing a dead zone mechanism and Signal-to-Noise Ratio loss to better capture asymmetric safety costs and improve constraint alignment.
Researchers propose Swap-guided Preference Learning (SPL) to address posterior collapse issues in Variational Preference Learning for RLHF systems. SPL introduces three new components to better capture personalized user preferences and improve AI alignment with diverse human values.
Researchers introduce the What Is Missing (WIM) rating system for Large Language Models that uses natural-language feedback instead of numerical ratings to improve preference learning. WIM computes ratings by analyzing cosine similarity between model outputs and judge feedback embeddings, producing more interpretable and effective training signals with fewer ties than traditional rating methods.
Researchers have developed a new preference learning framework that addresses bias in AI alignment by ensuring policies reflect true population distributions rather than just majority opinions. The approach uses social choice theory principles and has been validated on both recommendation tasks and large language model alignment.
Researchers developed AWARE-US, a system to improve AI agents' ability to handle failed database queries by intelligently relaxing the least important user constraints rather than simply returning 'no results'. The system uses three LLM-based methods to infer constraint importance from dialogue, achieving up to 56% accuracy in correct constraint relaxation.
Researchers conducted a cross-modal study comparing human preference annotations between text and audio formats for AI alignment. The study found that while audio preferences are as reliable as text, different modalities lead to different judgment patterns, with synthetic ratings showing promise as replacements for human annotations.
Researchers used eye-tracking to analyze how humans make preference judgments when evaluating AI-generated images, finding that gaze patterns can predict both user choices and confidence levels. The study revealed that participants' eyes shift toward chosen images about one second before making decisions, and gaze features achieved 68% accuracy in predicting binary choices.
Researchers developed CMA-ES-IG, a new algorithm that helps robots learn user preferences more effectively by incorporating user experience considerations. The algorithm suggests perceptually distinct and informative robot behaviors for users to rank, showing improved scalability, computational efficiency, and user satisfaction compared to existing methods.
