Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions

Researchers have developed a diagnostic evaluation framework using Construction Grammar to test whether large language models like GPT-o1 can truly understand language semantics beyond memorized patterns. The study reveals that state-of-the-art models fail to generalize across syntactically identical constructions with different meanings, dropping over 40% in performance on this task—a capability humans perform intuitively.

This research addresses a fundamental limitation in how we evaluate large language models. While LLMs demonstrate impressive performance on many benchmarks, the scale of pretraining data creates ambiguity about whether models have genuinely learned linguistic principles or merely memorized patterns. The authors leverage Construction Grammar, a psycholinguistically grounded framework, to probe whether models understand abstract syntactic-semantic relationships the way humans do.

The core contribution lies in distinguishing between performance on common linguistic patterns and generalization to novel combinations. Phrasal constructions—like "take advantage" versus "take a break"—share syntactic structure but carry distinct semantic meanings. Humans effortlessly abstract over this structure to understand new instantiations, yet the evaluation shows GPT-o1 and comparable models struggle dramatically when constructions are syntactically identical but semantically divergent.

This finding has significant implications for AI development and deployment. If models cannot reliably parse constructional semantics, their understanding remains brittle and potentially unreliable in edge cases or creative language use. This matters for applications requiring nuanced comprehension, from content moderation to technical writing assistance. The 40% performance drop is not marginal—it suggests a gap between apparent and actual linguistic competence that developers and organizations should consider.

The public release of the evaluation dataset creates an opportunity for researchers to systematically improve model architectures. Future work should investigate whether architectural changes, training approaches, or scale improvements can better capture constructional semantics. This represents a shift toward more rigorous evaluation methodologies that expose real limitations rather than inflating benchmark performance.

• →GPT-o1 and state-of-the-art LLMs show a 40% performance drop when distinguishing syntactically identical constructions with different meanings.
• →Construction Grammar provides a psycholinguistically valid framework for evaluating genuine semantic understanding versus memorization.
• →Current LLMs fail to generalize constructional semantics the way humans intuitively do, revealing a fundamental gap in linguistic competence.
• →The newly released evaluation dataset enables systematic assessment of out-of-domain language generalization in large language models.
• →Performance gaps on constructional semantics suggest limitations in model reliability for applications requiring nuanced language understanding.