Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries

Researchers introduce ICR (Inductive Conceptual Rating), a new qualitative metric for evaluating meaning in large language model text summaries that goes beyond simple word similarity. The study found that while LLMs achieve high linguistic similarity to human outputs, they significantly underperform in semantic accuracy and capturing contextual meanings.

• →New ICR metric combines semiotics and hermeneutics to better evaluate meaning in LLM-generated text summaries.
• →LLMs show high linguistic similarity but poor semantic accuracy compared to human-generated summaries across five datasets.
• →Performance gaps highlight fundamental differences between statistical approximation and human interpretive meaning.
• →LLM performance improves with larger datasets but remains inconsistent across different models.
• →Study advocates for qualitative evaluation frameworks when assessing meaning in AI-generated content.