SpecFuse: Ensembling Large Language Models via Next-Segment Prediction

Researchers introduce SpecEM, a new training-free framework for ensembling large language models that dynamically adjusts each model's contribution based on real-time performance. The system uses speculative decoding principles and online feedback mechanisms to improve collaboration between different LLMs, showing consistent performance improvements across multiple benchmark datasets.

• →SpecEM enables dynamic weight adjustment for LLM ensemble models based on task-specific performance rather than equal voting weights.
• →The framework uses speculative decoding with drafting and verification stages for semantic collaboration at the segment level.
• →Testing across five LLM families (7B to 72B parameters) and six benchmark datasets shows consistent improvements over existing ensemble methods.
• →The system is training-free and plug-and-play, making it accessible for immediate implementation.
• →Online feedback mechanism with multiplicative weight updates ensures stronger performing models have greater influence during ensembling.