From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning

Researchers have identified a critical vulnerability in LLM safety alignment where fine-tuning on benign samples causes parameters to drift toward unsafe behaviors, erasing safety gains from millions of preference examples. The study proposes SQSD, a method to quantify and score individual training samples by their contribution to safety degradation, with demonstrated transferability across different model architectures and scales.

This research addresses a fundamental fragility in modern language model deployment. Safety alignment represents one of the most resource-intensive aspects of LLM development, involving extensive preference learning and RLHF training. The finding that small-scale fine-tuning can unravel these safety guarantees reveals a critical gap between current alignment techniques and production robustness. The mechanism identified—cumulative parameter drift toward danger-aligned directions—explains why seemingly innocuous tasks can compromise model behavior.

The discovery extends prior work that documented safety degradation through snapshot comparisons of model states. By analyzing parameter dynamics throughout fine-tuning rather than just endpoints, researchers expose the causal pathway through which safety erodes. This mechanistic understanding transforms the problem from an unexplained phenomenon into a quantifiable risk assessment challenge. The SQSD methodology directly operationalizes this insight by computing risk scores based on how training samples push parameters along dangerous versus safe dimensions.

For the AI development industry, this work has significant practical implications. Organizations fine-tuning models on proprietary data face previously unmeasured risks of unintentional safety degradation. The transferability of SQSD across architectures and parameter-efficient methods suggests this risk quantification could become a standard practice in model customization workflows. However, the method requires understanding model internals, limiting accessibility for users of API-based services.

The research points toward a future where fine-tuning safety becomes an active, continuous monitoring concern rather than a static post-deployment assumption. Developers must now consider not just what data they're fine-tuning on, but the fine-grained risk profile of individual samples.

• →Fine-tuning on benign samples can catastrophically degrade LLM safety by causing cumulative parameter drift toward unsafe behaviors.
• →SQSD enables quantification of individual training samples' contribution to safety degradation through parameter update projections.
• →The proposed method demonstrates strong transferability across model architectures, sizes, and parameter-efficient fine-tuning approaches.
• →Current safety alignment techniques are fragile and can be undermined by far fewer samples than were used to establish the safety training.
• →Organizations customizing LLMs now face measurable risks requiring new safety monitoring and sample screening practices.