Exploring Information Seeking Agent Consolidation

Researchers present the first systematic study consolidating specialized information-seeking agents into a single foundation model, comparing data-level mixing with parameter-level merging across 26 methods and 10 benchmarks. Parameter-level merging achieves comparable performance to data mixing at significantly lower training cost while better preserving out-of-domain capabilities, offering practical efficiency gains for cross-domain AI deployment.

This research addresses a fundamental challenge in AI systems: the fragmentation of specialized information-seeking agents across different domains. Currently, separate models excel at retrieving information from the open web, documents, or local knowledge bases, creating inefficiencies and limiting scalable deployment. The study evaluates whether consolidating these capabilities into unified models is feasible and desirable.

The research compares two architectural approaches with meaningful practical differences. Data-level mixing trains a single model on combined datasets from multiple sources, while parameter-level merging takes independently trained specialist models and fuses them in parameter space. This distinction matters because parameter merging could potentially preserve specialized expertise while achieving generalization. The empirical evaluation of 26 merging methods across 10 benchmarks provides unprecedented breadth, introducing new metrics like Composite Score and Imbalance Score to compare heterogeneous performance fairly.

Key findings suggest parameter merging offers substantial advantages: it matches data mixing's performance while requiring dramatically less compute, operates regardless of expert order, and critically, preserves capabilities outside the training domain—a capability data mixing universally loses. This preservation of out-of-domain knowledge indicates parameter merging maintains model robustness better than brute-force retraining approaches.

For the AI development community, these insights have practical implications. Organizations deploying multi-domain information systems could reduce infrastructure costs significantly by merging trained experts rather than retraining unified models. The method-selection guide and design principles the authors distill offer actionable guidance for practitioners. However, generalization to other agent types and real-world performance under production constraints remain open questions worth monitoring.

• →Parameter-level merging achieves competitive performance with data mixing at a fraction of training cost.
• →Parameter merging preserves out-of-domain capabilities that data mixing approaches consistently lose.
• →Study evaluates 26 merging methods across 10 benchmarks, establishing empirical best practices for consolidation.
• →Cross-scenario stability directly correlates with consolidation quality, providing measurable consolidation metrics.
• →Findings enable cost-efficient deployment of multi-domain information-seeking agents through expert parameter fusion.