Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

Researchers challenge the effectiveness of the MLLM-CL benchmark for continual learning in multimodal AI models, demonstrating that a simple routing method matches complex MLLM-based approaches while requiring far fewer resources. The study reveals fundamental limitations in the benchmark's design that favor isolated learning over genuine continual transfer, prompting calls for more rigorous evaluation frameworks.

This research exposes critical weaknesses in how the AI community evaluates continual learning for multimodal large language models. The authors demonstrate that MR-LoRA, previously considered state-of-the-art, relies on unnecessarily complex architecture assumptions. Their proposed RePRo method achieves comparable performance using only frozen pretrained features and task prototypes, achieving substantial computational savings without sacrificing accuracy. This finding suggests the field may have over-engineered solutions to problems that simpler methods can adequately solve.

The deeper issue identified involves structural flaws in MLLM-CL itself. The benchmark's highly separable task representations in feature space mean that models can succeed by learning tasks in isolation rather than developing genuine continual learning capabilities. Combined with a fixed task curriculum, evaluation results become sensitive to specific ordering rather than reflecting robust performance across diverse learning trajectories. These constraints significantly limit the benchmark's ability to assess real-world continual adaptation scenarios where tasks overlap and arrive in unpredictable sequences.

This work has implications for AI research methodology and resource allocation. Teams investing in increasingly sophisticated architectures for continual learning may be addressing benchmark artifacts rather than fundamental problems. The proposed improvements—overlapping task manifolds, randomized task orders, fine-grained domain shifts, and forward-transfer metrics—would create more representative evaluation conditions. For practitioners deploying MLLMs in production environments with evolving domains, this research highlights that current benchmarks may not adequately predict real-world performance. The findings encourage the community to prioritize benchmark design rigor alongside architectural innovation.

• →Simple training-free routing methods match complex MLLM-based routers while reducing computational overhead significantly
• →MLLM-CL benchmark tasks are too separable, rewarding isolation learning instead of genuine continual transfer
• →Fixed task ordering in current benchmarks makes results sensitive to curriculum choice rather than reflecting robust learning
• →Shared expert architectures provide no measurable benefit in continual MLLM learning despite theoretical appeal
• →Future benchmarks need overlapping tasks, multiple orderings, and forward-transfer metrics to properly evaluate continual learning