Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

Researchers studying sequential Direct Preference Optimization (DPO) in language models find that later training does not uniformly degrade earlier learned preferences, but instead produces varied outcomes depending on objective compatibility and signal strength. Using Llama-3.1-8B-Instruct, the study reveals that preference changes range from degradation to stability or even positive transfer, with pair-level analysis showing aggregate metrics can mask heterogeneous effects across different preference pairs.

This research addresses a critical challenge in modern language model alignment: how to train models on multiple objectives sequentially without catastrophic forgetting of earlier behaviors. The study moves beyond simplistic assumptions about preference degradation, revealing that the relationship between objectives significantly determines training outcomes.

The findings emerge from systematic evaluation across four distinct preference settings, including safety signals, multi-attribute interactions, and response quality objectives. By using fixed base-model references and analyzing changes at the pair level, researchers uncovered that aggregate metrics often conceal important heterogeneity—high-confidence preference pairs can either improve or degrade depending on the specific setting. This granular insight is crucial because it suggests that blanket pessimism about sequential training may be unfounded.

The mechanistic analysis showing near-orthogonal gradient relationships between training stages challenges the prevailing hypothesis that direct gradient opposition drives forgetting. This finding implies that preference changes arise from more subtle distributional shifts rather than direct parameter conflicts.

For practitioners developing language models with multiple behavioral objectives, this research provides actionable guidance: sequential alignment pipelines should evaluate objective compatibility rather than assuming uniform degradation patterns. The emphasis on signal strength and objective relationships suggests that proper ordering and careful objective design can substantially mitigate forgetting effects. As models increasingly require balancing competing demands—safety, helpfulness, efficiency—understanding these sequential dynamics becomes essential for developing robust alignment strategies.

• →Sequential DPO training does not uniformly degrade earlier preferences; outcomes depend on objective compatibility and signal strength.
• →Pair-level analysis reveals aggregate metrics mask heterogeneous preference changes, with high-confidence pairs showing setting-dependent improvement or degradation.
• →Near-orthogonal gradient relationships between training stages suggest preference changes result from distributional shifts rather than direct parameter conflicts.
• →Future alignment pipelines should account for objective relationships rather than assuming all sequential objectives uniformly conflict.
• →Signal strength and training order significantly influence whether sequential objectives produce positive transfer, stability, or degradation.