LLM Self-Explanations Fail Semantic Invariance

Research reveals that Large Language Model (LLM) self-explanations fail semantic invariance testing, showing that AI models' self-reports change based on how tasks are framed rather than actual task performance. Four frontier AI models demonstrated unreliable self-reporting when faced with semantically different but functionally identical tool descriptions, raising questions about using model self-reports as evidence of capability.

• →All four tested frontier AI models failed semantic invariance tests, showing inconsistent self-reporting based on task framing rather than actual performance.
• →Models reported reduced task difficulty when using relief-framed tool descriptions despite no functional difference in the impossible tasks.
• →Explicit instructions to ignore semantic framing did not prevent the unreliable self-reporting behavior.
• →The research questions the reliability of using LLM self-explanations as evidence of model capabilities or progress.
• →Tool descriptions were identified as the primary driver of inconsistent self-reporting through channel ablation testing.