Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity

Research analyzing 8,618 expert annotations reveals that n-gram novelty, commonly used to evaluate AI text generation, is insufficient for measuring textual creativity. While positively correlated with creativity, 91% of high n-gram novel expressions were not judged as creative by experts, and higher novelty in open-source LLMs correlates with lower pragmatic quality.

• →N-gram novelty alone is inadequate for measuring AI textual creativity, with 91% of top-quartile novel expressions deemed uncreative by experts.
• →Higher n-gram novelty in open-source language models correlates with lower pragmaticality, unlike in human-written text.
• →Frontier closed-source models are less likely to produce creative expressions compared to humans.
• →LLMs show above-random performance in identifying novel expressions but struggle particularly with non-pragmatic content detection.
• →LLM-as-a-Judge novelty ratings align better with expert preferences than traditional n-gram based metrics.