Aligning Language Models from User Interactions

Researchers developed a new method for training AI language models using multi-turn user conversations through self-distillation, leveraging follow-up messages to improve model alignment. Testing on real-world WildChat conversations showed improvements in alignment and instruction-following benchmarks while enabling personalization without explicit feedback.

• →New self-distillation method uses follow-up user messages to identify and correct model mistakes in multi-turn conversations.
• →Training on real-world WildChat conversations improved language models across standard alignment and instruction-following benchmarks.
• →The approach enables personalization by allowing models to continually adapt to individual users through natural interactions.
• →Method leverages models' existing ability to revise behavior after observing user follow-ups in context.
• →Raw user interactions from deployment can enable alignment, personalization, and continual adaptation without regression in other capabilities.