Probing Routing-Conditional Calibration in Attention-Residual Transformers

Researchers question whether routing traces in Attention-Residual transformers provide genuine evidence of improved post-hoc calibration beyond standard confidence metrics. Through rigorous statistical testing with matched controls, the study finds that routing-specific features offer minimal stable evidence of better calibration, suggesting previous claims of calibration improvements may reflect methodological artifacts rather than true model improvements.

This paper addresses a critical methodological gap in how modern transformer architectures are evaluated. Attention-Residual transformers have been accompanied by claims that their internal routing mechanisms provide calibration-relevant uncertainty signals. The authors challenge this by asking whether routing traces genuinely improve post-hoc calibration or simply create an illusion through inadequate experimental controls.

The research employs a rigorous diagnostic approach, stratifying examples by routing-derived states and comparing subgroup gaps against permutation nulls. Critically, they include matched controls—confidence-only baselines and capacity-matched MLPs—to isolate routing-specific contributions from confounding factors. This methodological rigor mirrors best practices in causal inference.

The findings are striking: scalar routing summaries show no stable calibration improvements, with only 1 of 30 permutation tests rejecting the null hypothesis. More sophisticated approaches like AR-CondCal fall within the variance band of simpler confidence baselines. Even full-vector MLPs over routing features lose their apparent advantage when tested against capacity-matched confidence-only controls, and shuffled routing profiles perform comparably.

This work matters because it prevents overstating model improvements based on incomplete evaluation. In machine learning systems deployed for high-stakes applications, distinguishing genuine advances from artifacts is essential. The paper demonstrates that claims of routing-aware calibration require careful validation including matched baselines, bandwidth sensitivity analysis, capacity controls, and permutation testing. The results suggest the AI community should adopt more stringent experimental standards when evaluating architectural innovations claiming uncertainty quantification benefits.

• →Routing traces in Attention-Residual transformers fail to provide stable evidence of improved calibration beyond confidence alone
• →Proper experimental controls including matched-confidence baselines are essential to avoid confounding routing features with statistical artifacts
• →Only 1 of 30 within-bin permutation tests showed significant routing-conditional miscalibration, indicating claims require stronger evidence
• →Full-vector MLP approaches appear to improve calibration but lose advantage when capacity-matched confidence-only controls are included
• →The study establishes methodological standards for evaluating internal-state calibration in modern architectures