#preference-data News & Analysis

3 articles tagged with #preference-data. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBearisharXiv – CS AI · May 277/10

🧠

Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories

Researchers found that LLM-generated stories suffer from severe lack of diversity, with just 11 specific words appearing in 88.3% of outputs across multiple models. These recurring elements—character names like Elias and Mara, settings like lighthouses, and professions like clockmaker—originate from preference data used in model alignment rather than training data, revealing how small datasets can disproportionately shape AI outputs.

AINeutralarXiv – CS AI · Apr 147/10

🧠

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

Researchers introduce WIMHF, a method using sparse autoencoders to decode what human feedback datasets actually measure and express about AI model preferences. The technique identifies interpretable features across 7 datasets, revealing diverse preference patterns and uncovering potentially unsafe biases—such as LMArena users voting against safety refusals—while enabling targeted data curation that improved safety by 37%.

AINeutralarXiv – CS AI · Jun 16/10

🧠

Reward Learning from Best-of-$N$ Preference Data: Targets, Tradeoffs, and Design Principles

Researchers analyze how Best-of-N sampling constructs preference data for reward learning in AI systems, deriving closed-form targets and identifying a fundamental tradeoff between margin and connectivity governed by N size. The work provides design principles for practitioners: use larger N when preference labels are scarce, smaller N when generation capacity is limited, and optimize base distributions to prioritize comparisons most relevant at deployment.