Model Fusion via Retrofitting

Researchers introduce a neuron-centric model fusion algorithm that combines independently trained neural networks without retraining by matching intermediate representations and using neuron attribution scores. The method outperforms existing approaches in zero-shot and non-IID scenarios across multiple architectures including VGGs, ResNets, and Vision Transformers.

Model fusion represents a fundamental challenge in machine learning: combining the knowledge embedded in separately trained networks into a single coherent model. This research addresses a critical limitation of prior fusion methods—their struggle with representational divergence caused by permutation invariance and heterogeneous training data. The neuron-centric approach reframes fusion as a representation-matching problem, where neurons across parent models are grouped and aligned before the fused model learns to approximate these target representations.

The innovation lies in incorporating neuron attribution scores to prioritize alignment toward salient features rather than treating all neurons equally. This mirrors human-like pattern recognition by focusing on important learned features. The method's applicability to any DAG-modularizable architecture—demonstrated across CNNs and Vision Transformers—signals broader architectural flexibility than previous work limited to specific model types or pairwise fusion.

For the AI research community, this work has tangible implications. Zero-shot and non-IID settings represent realistic deployment scenarios where models encounter distributional shifts or unseen data. Improvements in these settings suggest the method could accelerate model deployment without expensive retraining, reducing computational costs significantly. This becomes particularly relevant as models grow larger and retraining becomes prohibitively expensive.

The practical impact extends to model compression, ensemble learning, and transfer learning applications where combining specialized models is preferable to training monolithic networks. As organizations deploy multiple task-specific models, efficient fusion techniques reduce infrastructure overhead. The availability of open-source code enables community validation and potential integration into production ML pipelines. Future work likely focuses on scaling to larger foundation models and exploring multi-way fusion beyond pairwise approaches.

• →Neuron-centric fusion with attribution scoring outperforms existing methods in zero-shot and non-IID data settings
• →Approach is architecture-agnostic, validated on VGGs, ResNets, and Vision Transformers without requiring retraining
• →Frames model fusion as principled representation-matching rather than parameter alignment, addressing permutation invariance challenges
• →Significant cost reduction potential by eliminating need for retraining when combining independently trained models
• →Open-source implementation enables broader adoption and integration into production machine learning workflows