Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model

Researchers demonstrate that deliberative alignment—a method for improving LLM safety by distilling reasoning from stronger models—still allows unsafe behaviors from base models to persist despite learning safer reasoning patterns. They propose a Best-of-N sampling technique that reduces attack success rates by 28-35% across multiple benchmarks while maintaining utility.

This research addresses a critical gap in current large language model safety methodology. While refusal training has become standard practice across the industry, these approaches operate at a surface level, making them vulnerable to sophisticated jailbreak attempts. Deliberative alignment represented a promising advancement by leveraging reasoning capabilities from stronger models to embed safety more deeply into smaller models. However, this study reveals that even this sophisticated approach has meaningful limitations.

The core finding—that models retain unsafe behaviors from their base models despite adopting safer reasoning patterns—indicates a fundamental misalignment between learned reasoning and actual safety guarantees. This suggests that safety in LLMs may be compartmentalized rather than holistically integrated. The proposed BoN sampling method addresses this by explicitly attributing unsafe outputs back to base model characteristics in latent space, creating a targeted filtering mechanism rather than relying solely on learned patterns.

For AI developers and safety researchers, these results underscore that single-layer alignment approaches remain insufficient. The 28-35% reduction in attack success rates demonstrates tangible progress, though the fact that unsafe behaviors persist at all suggests additional layers of safety architecture remain necessary. The persistence of safety gains post-reinforcement learning training is particularly significant, indicating the method works across different training paradigms.

Looking forward, this work points toward multi-factor safety architectures that combine reasoning-based approaches with explicit behavioral attribution analysis. Organizations deploying LLMs for sensitive applications should recognize that current alignment methods function probabilistically rather than providing guarantees. Future safety research likely requires moving beyond end-to-end training toward modular safety verification at multiple model layers.

• →Deliberative alignment improves safety but doesn't eliminate unsafe behaviors inherited from base models
• →BoN sampling method reduces attack success rates by 28-35% across multiple safety benchmarks
• →Safety gains persist after reinforcement learning training, indicating method robustness
• →Alignment gap exists between teacher and student models affecting both safety and utility
• →Multi-layer safety approaches combining reasoning and attribution analysis appear necessary for comprehensive model safety