When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis

Researchers introduce 'fragility' as a complementary metric to linear probing for analyzing large language model pre-training, addressing the limitation that probe accuracy saturates early in training and becomes insensitive to ongoing representational changes. By measuring activation noise tolerance levels, fragility reveals structural evolution in how models encode lexical versus compositional information across layers, demonstrating that data curation and architectural choices leave distinct signatures invisible to traditional accuracy metrics.

Linear probing has become a standard tool for understanding what information language models encode during pre-training, but the technique suffers from a critical blind spot: accuracy metrics plateau within the first few thousand training steps, rendering the majority of pre-training opaque to analysis. This research addresses that limitation by proposing fragility—a measure of how resistant learned representations are to noise-induced perturbations—as a complementary diagnostic instrument. Where accuracy provides only binary insight into whether information is separable, fragility captures nuances about margin robustness and representational redundancy that continue evolving throughout training.

The work demonstrates that fragility reveals previously invisible patterns in representation learning. The researchers observe a clear lexical-to-compositional gradient in moral reasoning tasks, with models initially developing surface-level lexical detection before building more sophisticated compositional encoding. Critically, they validate compositional encoding by showing it transfers across constructions sharing no overlapping tokens, proving the representations capture genuine structural understanding rather than shallow pattern matching. Additionally, fragility detects that layer-depth robustness develops monotonically across training while accuracy metrics flatline, and distinguishes between fine-tuning approaches that produce identical probing accuracy but different noise-resilience profiles.

For the AI research community, this work establishes a more sensitive diagnostic framework for understanding pre-training dynamics. It suggests that traditional probing metrics may fundamentally mischaracterize model learning phases, and that representation quality involves dimensions beyond task-separability. This has implications for model interpretability research and could inform architectural or training decisions aimed at building more robust representations.

• →Fragility metric reveals representation evolution invisible to traditional probe accuracy, which saturates early in training.
• →Lexical moral reasoning emerges before compositional encoding, suggesting distinct learning phases with different robustness signatures.
• →Identical probing accuracy across different fine-tuning approaches masks distinct fragility fingerprints, indicating data curation shapes robustness independently.
• →Layer-depth robustness develops monotonically while accuracy remains flat, exposing accuracy's insensitivity to ongoing representational refinement.
• →Fragility transfers across different construction types with no shared tokens, validating compositional encoding as genuine structural understanding rather than token memorization.