Activation Differences Reveal Backdoors: A Comparison of SAE Architectures

Researchers demonstrate that Differential SAEs (Diff-SAE) significantly outperform Crosscoders in detecting backdoor attacks in language models, achieving a 0.40 Backdoor Isolation Score with perfect precision. The study reveals that backdoors manifest as directional activation shifts rather than sparse features, providing critical insights for AI safety monitoring and interpretability tool development.

This research addresses a critical vulnerability in modern language models: the ability to embed hidden backdoors that trigger harmful behavior under specific conditions. The team's controlled experiment using SQL injection attacks demonstrates a fundamental difference in how two interpretability approaches detect these threats. While Crosscoders rely on identifying sparse feature activations, Diff-SAE's approach of analyzing directional shifts in activation patterns proves dramatically more effective, suggesting that backdoors leave distinctive directional signatures in model behavior rather than isolated sparse features.

The gap between the two architectures has profound implications for AI safety infrastructure. A Backdoor Isolation Score of 0.40 with 1.0 precision means Diff-SAE reliably identifies relevant features without false positives, while Crosscoders' sub-0.02 scores indicate near-complete failure. This performance holds across multiple layers and fine-tuning approaches, indicating the finding's robustness. The particular strength of Diff-SAE with full-rank fine-tuning suggests that understanding how models distribute learned behaviors across their parameter space remains crucial.

For AI developers and safety teams, this research provides actionable guidance on which interpretability tools offer genuine detection capabilities. Organizations deploying large language models now have evidence that certain mechanistic interpretability approaches fundamentally cannot detect particular attack classes. The implication extends beyond detection: understanding that backdoors operate through directional shifts suggests that adversarial training and safety measures should focus on activation space dynamics rather than isolated neuron behavior.

Looking forward, researchers should investigate whether other attack classes follow similar directional patterns, and whether this insight extends to other model architectures beyond SmolLM2-360M. The field needs standardized benchmarks for interpretability tool effectiveness against various attack types.

• →Diff-SAE achieves 0.40 BIS with perfect precision while Crosscoders fail with sub-0.02 scores, demonstrating superior backdoor detection capability
• →Backdoors manifest as directional activation shifts rather than sparse feature activations, explaining why difference-based representations outperform feature-based approaches
• →Performance gap persists across multiple transformer layers and both LoRA and full-rank fine-tuning regimes, indicating robust findings
• →Full-rank fine-tuning produces cleaner backdoor signals than LoRA, suggesting different attack manifestation patterns across fine-tuning methods
• →These findings necessitate reevaluation of existing interpretability tools used for AI safety monitoring and model auditing