Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation

Researchers introduce M-JudgeBench, a comprehensive benchmark for evaluating Multimodal Large Language Models (MLLMs) used as judges, and propose Judge-MCTS framework to improve judge model training. The work addresses systematic weaknesses in existing MLLM judge systems through capability-oriented evaluation and enhanced data generation methods.

• →M-JudgeBench provides a ten-dimensional capability-oriented benchmark to assess MLLM judgment abilities across various evaluation scenarios.
• →Existing MLLM-as-a-judge systems show systematic weaknesses that current benchmarks fail to capture effectively.
• →Judge-MCTS framework generates pairwise reasoning trajectories to create better training data for judge models.
• →M-Judger models trained with the new framework demonstrate superior performance on both existing and new benchmarks.
• →The research establishes more principled foundations for evaluating and training AI judge models across domains.