From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models
Researchers systematically analyzed how eight large language models encode essay quality information in their hidden representations across three datasets. Using linear probing and neuron-level analysis, they found that essay quality is encoded in linearly accessible form, emerges progressively across layers, and partially transfers across different essay prompts, with individual 'essay scoring neurons' showing strong correlation to scores.
This research addresses a critical gap in understanding how large language models perform automated essay scoring, a task increasingly deployed in educational technology. The study moves beyond treating LLMs as black boxes by empirically demonstrating that essay quality representations follow predictable, interpretable patterns within neural networks. Researchers discovered that linear probing—a simpler analytical technique—performs nearly as well as nonlinear probes, indicating that essay quality information is already efficiently encoded in ways accessible through straightforward mathematical operations.
The findings build on broader efforts to interpret deep learning systems and validate that LLMs develop meaningful internal structures aligned with human-evaluated quality dimensions. This work extends interpretability research from vision and language understanding tasks into the specialized domain of automated assessment. The progressive emergence of essay quality signals across layers suggests that information processing in LLMs follows hierarchical patterns, with early layers capturing basic features and deeper layers refining quality assessments.
For educational technology developers and institutions deploying LLM-based scoring systems, these insights provide confidence that models operate through comprehensible mechanisms rather than opaque statistical correlations. The partial transferability across essay prompts indicates that learned quality representations generalize reasonably well, though rubric-specific variations require attention. The discovery of layer-sensitive behavior based on essay length highlights that model behavior adapts to input characteristics in structured ways. This interpretability research ultimately strengthens the case for responsible deployment of automated scoring systems by demonstrating that model decisions correlate with explainable internal features rather than spurious patterns.
- →Essay quality information is encoded linearly within LLM representations, making model decisions more interpretable and trustworthy
- →Individual neurons strongly correlate with essay scores, enabling targeted interventions and deeper understanding of scoring mechanisms
- →Quality representations emerge progressively across network layers and partially transfer across different essay prompts
- →Longer essays rely more heavily on deeper layers, suggesting adaptive processing based on input characteristics
- →Linear probes nearly match nonlinear probes in performance, indicating efficiently organized internal representations