ObfusQAte: A Proposed Framework to Evaluate LLM Robustness on Obfuscated Factual Question Answering

Researchers introduce ObfusQAte, a new framework to test Large Language Model robustness when faced with obfuscated or disguised factual questions. The study reveals that LLMs tend to fail or generate hallucinated responses when confronted with increasingly complex variations of questions across three dimensions of obfuscation.

• →ObfusQAte is the first comprehensive framework designed to evaluate LLM robustness on obfuscated factual question-answering.
• →The framework tests LLMs across three dimensions: Named-Entity Indirection, Distractor Indirection, and Contextual Overload.
• →LLMs show significant vulnerabilities when presented with nuanced variations of questions, often failing or hallucinating responses.
• →The research addresses a critical gap in understanding LLM limitations beyond standard benchmarking scenarios.
• →The ObfusQAte framework has been made publicly available to foster further research in this area.