PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

Researchers challenge the conventional wisdom that adapter interference in language models stems from parameter-space geometry by testing whether orthogonal or directionally independent updates reduce cross-domain interference. Their findings using DoRA-RBAC on multiple LLMs show geometry-aware merging provides no consistent advantage, suggesting interference mechanisms operate in shared nonlinear representations rather than linear parameter space.

The research addresses a fundamental challenge in scaling large language models across multiple domains: how to enable specialized behavior for different tasks without retraining or degrading performance. The adapter composition problem has become increasingly relevant as organizations deploy LLMs across varied use cases, from specialized QA benchmarks to safety-critical applications. The dominant theoretical framework suggested that parameter-space geometry—specifically orthogonality and directional independence—could predict and prevent interference when composing multiple domain-specific adapters.

This study systematically tests that hypothesis through DoRA-RBAC, a hierarchical framework combining weight-decomposed low-rank adaptation with geometry-aware merging strategies. The researchers evaluated performance across diverse benchmarks (GPQA, PubMedQA, SimpleQA, WMDP) on popular open-source models (LLaMA-3.1-8B, Mistral-7B). Contrary to expectations, geometry-aware Riemannian-inspired merging—theoretically superior to conventional Euclidean approaches—showed no consistent improvement over standard averaging. Angular alignment and orthogonality metrics proved weak predictors of actual composition performance.

These findings carry significant implications for LLM deployment and model architecture design. They suggest that current geometric approaches to understanding adapter interference oversimplify the underlying mechanisms, which likely involve complex interactions in higher-dimensional nonlinear representation spaces. For practitioners, this means optimizing adapters requires deeper investigation into representational dynamics rather than relying on parameter-space geometry heuristics. The work opens new research directions into understanding how multiple specialized models interact at the representational level, potentially requiring novel approaches to multi-domain adaptation beyond current geometric frameworks.

• →Geometry-aware merging strategies provided no consistent performance advantage over standard averaging in multi-domain adapter composition
• →Angular alignment and orthogonality metrics proved poor predictors of how well adapted models compose across domains
• →Adapter interference mechanisms operate primarily in shared nonlinear representations rather than linear parameter space
• →Current theoretical frameworks for understanding adapter composition may be oversimplified and require deeper investigation
• →DoRA-RBAC single-domain performance matches LoRA while maintaining modularity, but multi-domain optimization requires different strategies