FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users

Researchers propose FSPO (Few-Shot Preference Optimization), a meta-learning algorithm that personalizes large language models using minimal user preference data. The approach uses synthetically generated preferences to train models that can quickly adapt to individual user preferences, achieving 87% performance on synthetic users and 70% on real human users in evaluation tasks.

FSPO addresses a fundamental challenge in LLM deployment: creating systems that adapt to individual user preferences without requiring extensive personalized training data. This research reframes preference optimization as a meta-learning problem, allowing models to infer personalized reward functions from just a few labeled examples. The breakthrough lies in the careful construction of synthetic preference datasets—the researchers generated over 1 million synthetic personalized preferences to enable effective transfer learning to real users.

The work reflects growing recognition that one-size-fits-all LLMs limit practical applications. Virtual assistants, content curation, and customer-facing systems require nuanced personalization to drive user engagement and satisfaction. Prior approaches either relied on expensive real-world preference collection or suffered from poor transfer to actual users. FSPO bridges this gap by identifying that synthetic data quality depends critically on diversity and self-consistency—insights applicable beyond this specific domain.

For the AI industry, this represents progress toward more scalable personalization without massive data collection overhead. The 70% human winrate validates that synthetic pretraining translates meaningfully to real-world contexts, though substantial room for improvement remains. This approach has immediate implications for developers building consumer-facing AI products, where personalization directly correlates with retention and satisfaction.

Future directions include scaling FSPO to larger model families, expanding domain coverage, and reducing the gap between synthetic and real-user performance. The meta-learning framework suggests broader applications in rapid adaptation tasks beyond preference modeling.

• →FSPO enables LLM personalization using few-shot learning from synthetic preference data, achieving practical performance levels
• →Over 1 million synthetically generated personalized preferences were used to train models that transfer to real users
• →Human evaluation shows 70% winrate on real users in open-ended QA, validating synthetic-to-real transfer feasibility
• →Data diversity and coherent self-consistency prove crucial for successful transfer from synthetic preferences to actual user preferences
• →The approach reduces reliance on expensive real-world preference data collection while maintaining competitive personalization performance