M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity

Researchers developed M-QUEST, a new benchmark for evaluating AI models' ability to understand and detect toxicity in internet memes. The framework identifies 10 key dimensions for meme interpretation and tests 8 open-source language models, finding that instruction-tuned models perform better but still struggle with pragmatic inference.

• →M-QUEST benchmark consists of 609 question-answer pairs across 307 memes to test AI toxicity detection capabilities.
• →The framework identifies 10 dimensions crucial for meme understanding including textual, visual, emotional, and toxicity assessment.
• →Current large language models show varying performance in toxic meme interpretation depending on their architecture.
• →Models with instruction tuning and reasoning capabilities significantly outperform others in meme comprehension.
• →Pragmatic inference questions remain the most challenging aspect for AI models to solve accurately.