Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning

Researchers propose BADIT, a novel approach to improve large language model training by decomposing shared parameters into orthogonal basic abilities, mitigating the cross-task interference problem that degrades performance in multi-task instruction-tuning. The method outperforms existing solutions on the SuperNI benchmark across 6 LLMs by maintaining parameter orthogonality through spherical clustering during training.

Multi-task instruction-tuning has become central to modern LLM development, enabling models to excel across diverse applications. However, this training paradigm introduces a fundamental challenge: conflicting gradients from different tasks corrupt shared parameters, degrading overall model performance. Existing mitigation strategies like task-specific neuron selection and mixture-of-experts architectures attempt to isolate task parameters but remain incomplete, as many parameters necessarily span multiple tasks.

The BADIT framework represents a conceptual shift in how researchers approach this problem. Rather than isolating parameters, it models LLMs as encoding orthogonal basic abilities—foundational cognitive components that any task can be expressed as combinations of. By decomposing parameters into high-singular-value LoRA experts and enforcing orthogonality through spherical clustering, BADIT prevents gradient conflicts from corrupting shared knowledge. This approach mirrors decomposition methods in mathematics and signal processing, suggesting LLM architecture may naturally align with orthogonal feature spaces.

For the AI industry, this research addresses a critical bottleneck in developing increasingly capable models. As practitioners scale instruction-tuning to hundreds of tasks, interference effects compound, limiting performance gains. BADIT's empirical success across multiple model architectures indicates broader applicability rather than a niche solution. Organizations developing multi-task LLMs—from cloud providers to AI labs—would benefit from understanding this mechanism.

The findings suggest future LLM development may increasingly focus on parameter efficiency and orthogonal decomposition rather than simply scaling model size. As competition intensifies around inference costs and training efficiency, methodologies that maximize performance from shared parameters become economically significant. Continued validation across larger model scales and diverse task distributions will determine whether orthogonal decomposition becomes standard practice.

• →BADIT decomposes LLM parameters into orthogonal basic abilities to eliminate cross-task interference in multi-task training.
• →The method uses spherical clustering of rank-1 LoRA components to maintain orthogonality and prevent gradient conflicts.
• →Empirical testing on SuperNI benchmark with 6 LLMs demonstrates BADIT outperforms existing state-of-the-art mitigation approaches.
• →Orthogonal parameter decomposition offers a more complete solution than parameter isolation strategies like mixture-of-experts.
• →Findings have practical implications for training efficient multi-task LLMs at scale with improved performance consistency.