Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs

Researchers introduce IRIS Benchmark, the first comprehensive evaluation framework for measuring fairness in Unified Multimodal Large Language Models (UMLLMs) across both understanding and generation tasks. The benchmark integrates 60 granular metrics across three dimensions and reveals systemic bias issues in leading AI models, including 'generation gaps' and 'personality splits'.

• →IRIS Benchmark is the first framework to synchronously evaluate fairness in both understanding and generation tasks for multimodal AI models.
• →The benchmark normalizes 60 granular fairness metrics across three dimensions: Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability.
• →Evaluation of leading UMLLMs revealed systemic phenomena like 'generation gap' and 'personality splits' in AI model behavior.
• →The framework addresses the 'Tower of Babel' problem where conflicting fairness metrics hinder unified AI evaluation paradigms.
• →The extensible benchmark can integrate evolving fairness metrics and provides diagnostics to guide optimization of AI fairness capabilities.