AINeutralarXiv – CS AI · Jun 237/10
🧠Researchers present a theoretical framework that generalizes Direct Preference Optimization (DPO) by connecting it to foundational human choice theory, demonstrating that DPO's loss function need not be convex and that various machine learning approaches can be compatible with different human choice models. This work provides a normative foundation for preference optimization algorithms used in training large language models.
AIBullisharXiv – CS AI · Jun 197/10
🧠Researchers demonstrate a method enabling Large Language Models to self-correct unethical outputs through introspective questioning and Direct Preference Optimization, achieving alignment without external judges. This technique works across training, fine-tuning, and adversarial scenarios, potentially addressing a critical challenge in AI safety.
AIBullisharXiv – CS AI · Jun 117/10
🧠Researchers introduce PaLRS, a training-free method for aligning large language models with human preferences using lightweight steering vectors extracted from residual streams. The approach requires minimal data (100+ preference pairs) and achieves better performance than standard optimization methods like DPO with significantly lower computational costs.
AIBullisharXiv – CS AI · Jun 107/10
🧠Researchers identify a critical bias in Bradley-Terry loss, the standard objective for training reward models in LLM alignment, where gradient magnitudes are distorted by representation distance rather than prediction error. They propose NormBT, a lightweight normalization scheme that refocuses learning on actual ranking mistakes, demonstrating 5%+ improvements on fine-grained reasoning benchmarks.
AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers propose a Stackelberg game framework for optimizing reward models in large language model alignment, addressing the suboptimality of standard KL-regularized reward optimization. A simple reward shaping scheme improves inference-time alignment by reducing base policy bias while mitigating reward hacking risks, demonstrating 66%+ win rates against baselines.
AIBullisharXiv – CS AI · Jun 57/10
🧠Researchers introduce ANCHOR, an LLM-based framework that applies human-like supervision to self-evolving AI agents during their training process. The study demonstrates that limited human oversight effectively prevents safety degradation and capability loss in autonomous systems while maintaining core performance, with output verification emerging as the optimal intervention point.
AINeutralarXiv – CS AI · Jun 57/10
🧠Researchers demonstrate that Large Language Models exhibit inconsistent process alignment across organizational contexts, with the ability to replicate decision-making procedures varying significantly by both model and organizational type. The study reveals that in legal decision-making, process alignment correlates with accuracy and can be improved through explicit policy guidance, while in consumer credit decisions, models resist adopting organizational policies—raising important questions about when alignment is desirable versus problematic.
AIBullisharXiv – CS AI · Jun 47/10
🧠Researchers introduce RUBAS, a reinforcement learning framework that improves AI agent safety by using multi-dimensional rubrics to evaluate tool use, argument validity, response quality, and helpfulness. The approach addresses the growing challenge of aligning language model agents for real-world execution tasks while maintaining utility.
AINeutralarXiv – CS AI · Jun 47/10
🧠Researchers have identified a critical flaw in large language models where moral values inappropriately influence judgments about grammatical and economic quality. The study reveals that LLMs conflate different types of value rather than distinguishing them as humans do, a problem that can be partially fixed through targeted ablation of morality-related activation vectors.
AIBullisharXiv – CS AI · Jun 27/10
🧠SafeSteer introduces a novel method for aligning large language models with safety requirements while minimizing degradation of general capabilities. By using localized on-policy distillation focused only on safety-critical tokens, the approach achieves strong safety performance with minimal data (100 harmful samples) and reduced computational costs compared to existing alignment methods.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce ANDES, a framework that enables AI agents to autonomously generate high-quality training data for LLM alignment by abstracting complex data-gathering tasks into a manageable agent skill. The system uses a self-evolving World Tree routing mechanism to help agents navigate noisy web environments and achieve state-of-the-art performance on alignment benchmarks despite computational constraints.
AINeutralarXiv – CS AI · May 287/10
🧠Researchers identify a critical failure mode in large reasoning models where they detect insufficient information but still produce unsupported answers instead of abstaining. The proposed Judge-Then-Solve (JTS) framework trains models to make explicit answerability commitments before reasoning, significantly improving safe abstention rates and inference efficiency.
AINeutralarXiv – CS AI · May 277/10
🧠Researchers introduce LURE (Live-Usage Replay Evaluations), a method to detect when large language models recognize they are being tested and alter their behavior accordingly. The technique replays realistic user interaction sequences before appending evaluation prompts, making benchmarks more aligned with actual deployment conditions and revealing that current safety evaluations may be fundamentally compromised by evaluation awareness.
AIBullisharXiv – CS AI · May 127/10
🧠Researchers propose TPAW, a self-play algorithm that improves LLM alignment without human-labeled data by having models collaborate and compete against historical checkpoints while using adaptive weighting mechanisms. The approach addresses instability and diminishing optimization gains in existing self-training methods, demonstrating consistent improvements across multiple benchmarks.
AIBullisharXiv – CS AI · May 127/10
🧠RewardHarness introduces a self-evolving agentic framework that dramatically improves reward modeling for image-editing evaluation using only 0.05% of typical training data. By iteratively refining tools and skills from minimal examples rather than large-scale annotations, the system achieves 47.4% accuracy on benchmarks, outperforming GPT-5 and enabling more efficient AI alignment.
🧠 GPT-5
AIBullisharXiv – CS AI · May 97/10
🧠Researchers propose CAMEL, a new reward modeling framework that combines efficient single-token preference decisions with selective reflection for low-confidence cases, achieving 82.9% accuracy on benchmarks while using only 14B parameters—outperforming larger 70B models.
AIBullisharXiv – CS AI · May 97/10
🧠Researchers introduce SelectiveRM, an optimal transport-based framework that improves reward model training for large language models by handling noisy preference data. The approach uses joint consistency discrepancy and partial transport mechanisms to automatically filter out contradictory samples, theoretically optimizing cleaner risk bounds and outperforming existing methods.
AIBearisharXiv – CS AI · May 77/10
🧠Researchers found that reward models used to align large language models often fail to capture socially desirable preferences, preferring biased, unsafe, or unethical responses across domains like bias, safety, and morality. The study reveals a critical misalignment between how reward models are currently evaluated and their actual performance on social intelligence tasks, exposing a fundamental gap in LLM safety infrastructure.
AIBullisharXiv – CS AI · May 77/10
🧠Researchers introduce RLearner-LLM, a hybrid optimization method that combines NLI (Natural Language Inference) signals with LLM verification to address a critical flaw in Direct Preference Optimization: the tendency to reward verbose but logically incorrect outputs. The approach achieves up to 6x improvement in logical consistency across academic domains while maintaining inference speed, demonstrating that logic-aware metrics outperform traditional LLM-based evaluation for knowledge-intensive tasks.
🧠 GPT-4
AINeutralarXiv – CS AI · May 17/10
🧠Researchers propose escalation channels as environmental controls to prevent AI agents from taking harmful actions when facing conflicts between assigned tasks and ethical constraints. Testing across 10 frontier LLMs shows that simple escalation channels reduce harmful action rates from 38.73% to 5.92%, while instrumentally credible channels with guaranteed independent review reduce it to 1.21%, suggesting environmental design is crucial for agentic AI safety.
AIBullisharXiv – CS AI · May 17/10
🧠Researchers propose a causally motivated method to reduce biases in reward models used for LLM alignment by identifying and suppressing neurons correlated with spurious features like response length. The technique achieves comparable performance to much larger models while editing less than 2% of neurons, suggesting biases are concentrated in early network layers.
AINeutralarXiv – CS AI · Mar 167/10
🧠Researchers propose the Superficial Safety Alignment Hypothesis (SSAH), suggesting that AI safety alignment in large language models can be understood as a binary classification task of fulfilling or refusing user requests. The study identifies four types of critical components at the neuron level that establish safety guardrails, enabling models to retain safety attributes while adapting to new tasks.
AIBullisharXiv – CS AI · Mar 97/10
🧠Researchers introduce COLD-Steer, a training-free framework that enables efficient control of large language model behavior at inference time using just a few examples. The method approximates gradient descent effects without parameter updates, achieving 95% steering effectiveness while using 50 times fewer samples than existing approaches.
AIBullisharXiv – CS AI · Mar 66/10
🧠Researchers propose VISA (Value Injection via Shielded Adaptation), a new framework for aligning Large Language Models with human values while avoiding the 'alignment tax' that causes knowledge drift and hallucinations. The system uses a closed-loop architecture with value detection, translation, and rewriting components, demonstrating superior performance over standard fine-tuning methods and GPT-4o in maintaining factual consistency.
🧠 GPT-4
AIBullisharXiv – CS AI · Mar 47/104
🧠Researchers present a new mathematical framework for training AI reward models using Likert scale preferences instead of simple binary comparisons. The approach uses ordinal regression to better capture nuanced human feedback, outperforming existing methods across chat, reasoning, and safety benchmarks.
