Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures

Researchers introduce Temporal Logit Observability (TLO), a training-free diagnostic tool that reveals how LLM jailbreak attacks unfold over time by analyzing logit patterns during decoding, rather than just whether attacks succeed. The method identifies that attacks with identical success rates actually follow different failure pathways, enabling better safety evaluation and early-stopping defenses that reduce successful jailbreaks by over 50%.

Traditional Attack Success Rate (ASR) metrics provide binary outcomes—attacks either succeed or fail—without explaining the mechanisms behind failures. This research addresses a critical gap in LLM safety evaluation by introducing Temporal Logit Observability, which monitors the compliance-refusal margin throughout the generation process. The approach treats safety failures as temporal phenomena with distinct patterns rather than singular events, revealing that structurally identical harmful outputs can result from fundamentally different attack vectors.

The work builds on growing concerns about LLM robustness as jailbreaking techniques become increasingly sophisticated. Researchers have long recognized that understanding failure mechanisms matters more than counting failures, yet most evaluation frameworks remain outcome-focused. TLO bridges this gap by extracting interpretable safety signals directly from model logits without requiring retraining, making it immediately applicable across existing systems.

For AI safety practitioners and alignment researchers, TLO offers actionable insights beyond academic interest. The method's ability to differentiate between attacks with identical ASR enables targeted defenses—the researchers demonstrate a simple early-stopping rule derived from TLO patterns that prevents over 50% of successful jailbreaks while maintaining performance on benign queries. This suggests safety interventions can become more precise by understanding temporal failure patterns rather than applying blanket restrictions.

The research indicates that safety evaluation frameworks need temporal awareness to match real deployment requirements. As LLMs become more capable, distinguishing between superficially similar failures becomes essential for robust alignment. Future work should explore whether these temporal patterns generalize across model architectures and whether more sophisticated defense strategies can exploit the geometric structure TLO reveals.

• →Temporal Logit Observability reveals that attacks with identical success rates follow distinct pathways through model decoding, making previously hidden failure mechanisms observable.
• →A simple early-stop rule derived from TLO patterns blocks over 50% of successful jailbreaks without triggering false positives on benign queries.
• →The method operates training-free by analyzing compliance-refusal margins in logits, enabling immediate application to existing aligned LLMs.
• →Safety evaluation frameworks should report temporal patterns of failures alongside outcome metrics to better understand and defend against sophisticated attacks.
• →The approach works across four different aligned LLMs and three jailbreak paradigms, suggesting broad applicability for AI safety research.