AIBullisharXiv – CS AI · 7h ago6/10
🧠
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
Researchers propose Orthogonal Subspaces for Robust model Merging (OSRM), a technique that addresses performance degradation when combining multiple LoRA-fine-tuned language models into single multi-task systems. By constraining LoRA subspaces prior to fine-tuning, the method reduces task interference while maintaining individual task accuracy and improving compatibility with existing merging algorithms.