Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization

Researchers developed a new mathematical framework called Curvature-Weighted Capacity Allocation that optimizes large language model performance by identifying which layers contribute most to loss reduction. The method uses the Minimum Description Length principle to make principled decisions about layer pruning and capacity allocation under hardware constraints.

• →The framework introduces curvature-adjusted layer gain as a metric that outperforms gradient-norm-based scores for identifying important model layers.
• →Two convex optimization programs are provided: one for capacity allocation and another for pruning, both with closed-form solutions.
• →The method offers provable optimality and generalization guarantees with O(δ²) transfer regret bounds.
• →Solutions can be computed efficiently in O(K log 1/ε) time using bisection methods.
• →The framework elevates layer-wise optimization from empirical heuristics to theoretically grounded methodology.