Boosting Large Language Models with Mask Fine-Tuning

Researchers introduce Mask Fine-Tuning (MFT), a novel approach that improves large language model performance by applying binary masks to optimized models without updating weights. The method achieves consistent performance gains across different domains and model architectures, with average improvements of 2.70/4.15 in IFEval benchmarks for LLaMA models.

• →MFT improves LLM performance by strategically masking parts of well-optimized models rather than updating weights.
• →The technique achieved average gains of 2.70/4.15 in IFEval benchmarks across LLaMA2-7B and 3.1-8B models.
• →MFT can be deployed on already well-trained models and is compatible with other optimization procedures.
• →The approach challenges the assumption that maintaining model structural integrity is essential for performance.
• →This extends masking operations beyond traditional network pruning into broader model capability enhancement.