RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

RaguTeam won SemEval-2026 Task 8 using a seven-model LLM ensemble with a GPT-4o-mini judge selector, achieving a conditioned harmonic mean of 0.7827 and significantly outperforming the baseline. The research demonstrates that model diversity across families, scales, and prompting strategies drives superior performance in multi-turn response generation tasks.

RaguTeam's first-place finish in SemEval-2026 Task 8 validates an emerging pattern in large language model optimization: heterogeneous ensembles with intelligent selection mechanisms outperform individual models, regardless of base capability. The winning system orchestrates seven distinct LLMs through a judge-based selection framework, where GPT-4o-mini evaluates and selects the best response per instance. This approach yielded a 22% improvement over the strongest baseline, demonstrating tangible value creation through architectural choices rather than raw model scale alone.

The broader context reflects AI's evolution from monolithic model deployment toward sophisticated ensemble strategies. As individual LLM capabilities plateau, the industry increasingly focuses on combining different architectures and prompting variants to achieve specialized performance gains. RaguTeam's introduction of Meno-Lite-0.1, a 7B domain-adapted model with strong cost-performance characteristics, signals practical interest in efficiency-oriented alternatives to massive models. This aligns with industry trends toward cost optimization and edge deployment without sacrificing output quality.

The research carries implications for developers building production AI systems. The emphasis on prompt diversity and model family variation suggests that practitioners should experiment across model ecosystems rather than defaulting to single vendors. The ablation studies proving ensemble superiority over individual components provide empirical justification for multi-model architectures in quality-critical applications. The public code release democratizes access to these techniques, potentially accelerating industry adoption of ensemble methods for faithful, reference-grounded generation tasks.

• →Seven-model ensembles with judge-orchestrated selection beat individual baseline models by 22% on generation tasks.
• →Diversity across model families, scales, and prompting strategies is essential for ensemble performance gains.
• →Meno-Lite-0.1, a 7B domain-adapted model, offers compelling cost-performance trade-offs for specialized applications.
• →Judge-based selection frameworks enable dynamic, per-instance optimization without requiring retraining.
• →Ablations confirm that ensemble performance consistently exceeds any single constituent model.