Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations

Researchers systematically tested linear probes used to detect deception in large language models, finding they achieve near-perfect accuracy on clean data but fail dramatically under distributional shifts. The study reveals deception is encoded through distributed multi-dimensional features rather than a single direction, and probe robustness can be recovered through style augmentation, indicating failures stem from narrow training distributions rather than fundamental architectural limitations.

This research addresses a critical gap in AI safety: the reliability of deception-detection metrics that researchers increasingly rely upon to evaluate LLM trustworthiness. While linear probes reported AUROC scores exceeding 0.96 on controlled benchmarks, their collapse under real-world distributional shifts raises serious questions about their practical utility. The paper's systematic pressure-testing across the Gemma model family (1B-27B parameters) provides crucial diagnostic insights rather than merely documenting failures.

The geometric analysis reveals why single-direction probes fail: deception doesn't manifest as a simple linear phenomenon but rather as distributed sub-threshold features across multiple dimensions. This discovery has profound implications for how researchers should approach deception detection in increasingly sophisticated language models. The finding that style-augmented probes recover near-perfect detection (0.979-0.983 AUROC) on unseen styles suggests the problem is fundamentally solvable through better training data rather than requiring architectural innovations.

For the AI safety community, this work establishes that inverse scaling patterns previously attributed to model size are actually artifacts of narrow training distributions. This reframing is critical because it shifts focus from pessimistic scaling laws toward data engineering solutions. The entropy-proxy hypothesis rejection eliminates one potential simple mechanism, suggesting deception detection requires sophisticated multi-dimensional analysis rather than shortcut metrics.

Looking forward, developers building trustworthiness evaluations should prioritize diverse training distributions and multi-dimensional probe architectures. This research underscores that robust AI safety metrics demand careful attention to distributional coverage, not just algorithmic sophistication. The implications extend beyond deception detection to other behavioral assessments of LLMs.

• →Linear deception-detection probes fail under distributional shift despite exceeding 0.96 AUROC on clean benchmarks, revealing fundamental robustness limitations.
• →Deception is encoded as distributed multi-dimensional features, not a single linear direction, requiring k≥5 dimensional probes for reliable detection.
• →Style-augmented training recovers near-perfect detection across model sizes, proving the inverse scaling pattern is a training-distribution artifact rather than architectural limitation.
• →Entropy-based proxies fail to explain deception encoding (max correlation 0.454), eliminating a potential shortcut mechanism for detection.
• →Probe fragility reflects distributional narrowness, making data engineering and augmentation the key path to robust deception detection rather than architectural changes.