Residual Stream Analysis of Overfitting And Structural Disruptions

Researchers identified that repetitive safety training data causes large language models to develop false refusals, where benign queries are incorrectly declined. They developed FlowLens, a PCA-based analysis tool, and proposed Variance Concentration Loss (VCL) as a regularization technique that reduces false refusals by over 35 percentage points while maintaining performance.

• →Safety training datasets with low token entropy and diversity lead to false refusals in LLMs, increasing from 63% to 84% as safety data proportion rises.
• →FlowLens, a new PCA-based tool, reveals that safety examples concentrate variance along fewer components, reducing representational smoothness.
• →Variance Concentration Loss (VCL) is proposed as an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals.
• →VCL reduces false refusal rates by over 35 percentage points while maintaining performance on benchmarks like MMLU and GSM8K.
• →The research addresses a critical challenge in AI safety by balancing helpful and harmless behavior in language models.