Evaluating Adjective-Noun Compositionality in LLMs: Functional vs Representational Perspectives

A research study reveals that large language models develop strong internal compositional representations for adjective-noun combinations, but struggle to consistently translate these representations into successful task performance. The findings highlight a significant gap between what LLMs understand internally and their functional capabilities.

• →LLMs reliably develop compositional representations for adjective-noun combinations in their internal states.
• →Despite having good internal representations, LLMs fail to consistently translate them into functional task success.
• →There is a striking divergence between task performance and internal model states in LLMs.
• →The study used both prompt-based functional assessment and representational analysis to evaluate compositionality.
• →Contrastive evaluation methods are essential for obtaining complete understanding of model capabilities.