Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

A benchmarking study reveals demographic bias in multimodal large language models used for face verification, testing nine models across different ethnicity and gender groups. The research found that face-specialized models outperform general-purpose MLLMs, but accuracy doesn't correlate with fairness, and bias patterns differ from traditional face recognition systems.

• →FaceLLM-8B, the only face-specialized model tested, significantly outperformed general-purpose multimodal LLMs on face verification tasks.
• →Bias patterns in MLLMs differ from traditional face recognition systems, with different demographic groups being most affected depending on the specific model and benchmark used.
• →The most accurate models are not necessarily the fairest in terms of demographic representation.
• →Models with poor overall accuracy can appear fair due to uniformly high error rates across all demographic groups.
• →The study tested nine open-source MLLMs ranging from 2B to 8B parameters across four ethnicity groups and two gender groups.