TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability

Researchers demonstrate that task-aware layer pruning improves model performance on out-of-distribution (OOD) data while providing no benefits for in-distribution data. The improvement occurs because pruning removes layers that distort the task-adapted geometric representation, realigning OOD inputs with the model's learned task geometry.

This research addresses a fundamental challenge in machine learning: why removing certain neural network layers can paradoxically improve performance on data the model hasn't seen during training. The TAPIOCA study extends prior work on task-aware pruning by providing both empirical and mechanistic explanations for when and why this counterintuitive phenomenon occurs.

The geometric interpretation offers significant theoretical insight into how neural networks process information differently across domains. When models encounter out-of-distribution inputs, the layerwise activation norms and representational distances diverge from the patterns learned on training data. Some layers amplify this divergence, effectively corrupting the model's internal task representation. By selectively removing these distorting layers, the model maintains consistency between its learned geometry and novel inputs, effectively improving generalization.

This finding has substantial implications for model deployment and efficiency. Current practice typically treats pruning as a compression technique to reduce computational cost, often sacrificing some accuracy. TAPIOCA reveals that pruning can instead enhance robustness to distribution shift, a critical concern for real-world applications where test conditions inevitably differ from training conditions. This reframes pruning from a necessary evil of compression into a potential defensive mechanism against domain adaptation failures.

The consistency across polynomial regression tasks and large language models suggests this principle generalizes broadly. Future work should examine whether these insights apply to other architectures and whether active selection of pruning targets can be optimized for specific anticipated distribution shifts. The causal evidence provided through controlled interventions strengthens confidence in the geometric explanation.

• →Task-aware pruning improves out-of-distribution accuracy but provides no in-distribution benefits, suggesting orthogonal optimization objectives
• →OOD inputs distort the geometric representation profile learned by models on in-distribution data through altered activation norms and pairwise distances
• →Pruning works by removing layers that amplify geometric distortions, realigning OOD representations with the model's task-adapted geometry
• →This mechanism operates consistently across different model scales and architectures including large language models
• →Pruning can serve as a robustness enhancement rather than purely a compression technique, with implications for deployment in domain-shifted environments