Energy-Based Transformers as Predictors of Reading Difficulty

Researchers demonstrate that energy-based transformers, a class of neural networks linked to associative memory models, effectively predict reading difficulty across multiple eye-tracking and reading-time studies. The energy measure outperforms traditional metrics like surprisal and attention entropy, suggesting a unified approach to modeling human language processing.

This research advances computational psycholinguistics by introducing energy-based transformers as a novel framework for understanding how humans process language. Energy-based models provide a mathematically principled connection to Hopfield networks and associative memory theory, offering theoretical grounding that complements existing transformer-based approaches. The finding that energy measures predict reading times across three independent corpora—Natural Stories, UCL eye-tracking, and UCL self-paced reading—demonstrates robust generalization beyond single datasets.

The work builds on decades of cognitive science research showing that reading difficulty correlates with processing load. Previous studies established surprisal (prediction error) and attention entropy (uncertainty in model attention patterns) as complementary predictors. This research shows energy captures both effects while potentially explaining their mechanisms more parsimoniously through a single unified metric.

For AI researchers and computational linguists, this suggests energy-based formulations may offer advantages for modeling cognitive phenomena that attention-only mechanisms miss. The demonstration that energy at a single layer captures the well-known object/subject asymmetry in relative clause processing—a fundamental result in psycholinguistics—validates the approach against established behavioral benchmarks.

Looking forward, this opens pathways for integrating associative memory principles into transformer architectures. Researchers should investigate whether energy-based measures improve language model interpretability or transfer learning. The connection to Hopfield networks could inspire hybrid architectures combining transformers with classical memory models, potentially advancing both cognitive modeling and practical NLP applications.

• →Energy-based transformers provide a unified metric for predicting reading difficulty that subsumes both surprisal and attention entropy effects
• →The energy measure demonstrates robust predictive power across three independent reading-time corpora without additional tuning
• →Energy-based models establish a formal connection between transformer language models and associative memory theory, bridging AI and cognitive science
• →A single energy layer captures the object/subject asymmetry in relative clause processing, a well-established psycholinguistic phenomenon
• →This framework suggests future hybrid architectures combining transformers with classical Hopfield-style memory could enhance language model interpretability