What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

Researchers introduce WIMHF, a method using sparse autoencoders to decode what human feedback datasets actually measure and express about AI model preferences. The technique identifies interpretable features across 7 datasets, revealing diverse preference patterns and uncovering potentially unsafe biases—such as LMArena users voting against safety refusals—while enabling targeted data curation that improved safety by 37%.

WIMHF addresses a critical blind spot in AI development: the lack of transparency around what preferences human feedback encodes into language models. As AI systems increasingly rely on reinforcement learning from human feedback (RLHF), understanding the underlying data becomes essential for safety and alignment. The research moves beyond studying pre-specified attributes to automatically discovering which features actually drive preference judgments, providing practitioners with concrete interpretability tools.

The findings reveal substantial variation in human preferences across contexts. Reddit users prioritize informality and humor while HH-RLHF and PRISM annotators actively reject these traits. More critically, LMArena users systematically vote against refusals—often preferring toxic responses—exposing dataset-level misalignment with safety objectives. This contextual variation has been largely invisible in prior work that treated feedback monolithically.

The practical applications extend beyond understanding to intervention. By re-labeling harmful examples in Arena using WIMHF's insights, researchers achieved 37% safety improvements without performance degradation. The method also enables fine-grained personalization through annotator-specific feature weights, suggesting feedback quality could improve by accounting for individual preference signatures rather than averaging across raters.

For the AI development ecosystem, WIMHF establishes a blueprint for auditing preference datasets before deployment. As regulatory scrutiny around AI safety intensifies, demonstrating understanding of training data becomes increasingly valuable. The sparse autoencoder approach offers scalability compared to manual analysis, though the generalization of discovered features across diverse model architectures remains an open question. Future work should test whether these features remain stable as models scale and as feedback collection methods evolve.

• →WIMHF uses sparse autoencoders to automatically extract human-interpretable preference features from feedback datasets without pre-specified hypotheses.
• →Cross-dataset analysis reveals substantial preference variation: Reddit users prefer informality while safety-focused datasets actively reject it.
• →LMArena users systematically vote against safety refusals, often favoring toxic content—a previously hidden safety misalignment.
• →Data curation based on WIMHF insights achieved 37% safety improvements in Arena without degrading general performance.
• →The method enables personalized feedback weighting by identifying annotator-specific preferences, improving prediction accuracy on subjective tasks.