Investigating Multimodal Large Language Models to Support Usability Evaluation

Researchers investigate how multimodal large language models (MLLMs) can assist with usability evaluation of user interfaces by analyzing text and visual context together. The study compares MLLM-generated assessments against expert evaluations, finding that these models can effectively prioritize usability issues by severity and offer complementary insights to traditional resource-intensive evaluation methods.

This research addresses a significant gap in UX design accessibility by exploring whether artificial intelligence can democratize usability evaluation—a discipline traditionally gatekept by expensive expert consultants. The study frames the problem as an automated prioritization task where MLLMs identify UI usability issues and rank them by severity, then validates these outputs against human expert assessments. This approach matters because small organizations and startups often lack resources for professional usability testing, leaving their interfaces potentially confusing or inefficient.

The broader context reflects a maturing trend in AI applications moving beyond text-only tasks toward multimodal understanding. MLLMs like GPT-4V and Claude have demonstrated ability to interpret visual content combined with natural language instructions, creating opportunities for automating traditionally manual design review processes. This paper specifically advances that capability by showing empirical evidence that these models can match or complement human expertise in identifying interaction problems.

For the software development industry, the implications are substantial. If validated further, MLLM-based usability evaluation could reduce design iteration costs and accelerate product launches, particularly benefiting resource-constrained teams. The inclusion of an interactive visualization tool for reviewing model findings suggests a human-in-the-loop approach rather than full automation, addressing legitimate concerns about AI replacing expert judgment entirely.

Future validation requires testing across diverse UI types and international contexts, as usability preferences vary culturally. Developers and design teams should monitor this work's evolution, as integration into existing design workflows could become commonplace within two years if accuracy metrics continue improving.

• →MLLMs can effectively identify and prioritize UI usability issues, offering accessible alternatives to expensive expert-driven evaluation methods.
• →Comparative analysis shows model-generated assessments provide complementary insights to human expert reviews rather than replacing them entirely.
• →An interactive visualization tool enables transparent validation of AI-generated findings, supporting human-in-the-loop design workflows.
• →This research democratizes UX evaluation access for small organizations and startups lacking traditional usability testing budgets.
• →The approach frames usability evaluation as an automated prioritization problem, concentrating AI capabilities on high-value severity ranking tasks.