Preference Packing: Efficient Preference Optimization for Large Language Models

Researchers propose 'preference packing,' a new optimization technique for training large language models that reduces training time by at least 37% through more efficient handling of duplicate input prompts. The method optimizes attention operations and KV cache memory usage in preference-based training methods like Direct Preference Optimization.

• →Preference packing reduces LLM training time by at least 37% by optimizing duplicate input prompt handling.
• →The technique works by reducing attention operations and decreasing KV cache memory usage during training.
• →It applies to preference-based training methods like reward models and Direct Preference Optimization (DPO).
• →The method can be combined with existing optimizations like batch sorting for up to 3.22x speedup.
• →Testing was conducted on both text-only and image-included datasets showing consistent improvements.