When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

Researchers conducted the first comprehensive analysis of open-source direct preference optimization (DPO) datasets used to align large language models, revealing significant quality variations. They created UltraMix, a curated dataset that's 30% smaller than existing options while delivering superior performance across benchmarks.

• →First systematic comparison of popular open-source DPO datasets including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs.
• →The Magpie framework was used to annotate samples for task category, input quality, and preference reward without human annotations.
• →Analysis revealed structural and qualitative discrepancies in reward margins across different datasets.
• →UltraMix dataset achieves better performance while being 30% smaller by removing noisy and redundant samples.
• →All annotations, metadata, and the curated mixture are publicly released to advance preference optimization research.