Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Researchers propose a sleep-like mechanism for transformer language models that periodically consolidates context into persistent fast weights, reducing the computational burden of long sequences. The method shifts heavy computation offline while maintaining fast inference speeds, showing significant improvements on reasoning tasks that standard transformers struggle with.

This research addresses a fundamental limitation of transformer-based language models: their quadratic scaling with context length. As these models handle increasingly complex, long-horizon reasoning tasks, maintaining full attention across extended contexts becomes computationally prohibitive. The proposed sleep-consolidation mechanism mimics biological memory consolidation, where models convert recent context into compressed, persistent representations during offline periods before clearing their cache.

The technical contribution involves integrating this mechanism with state-space model (SSM) blocks, which naturally support efficient recurrent processing. During inference, models perform multiple offline passes over accumulated context to update fast weights—a computationally intensive process offloaded from the critical inference path. This architectural choice elegantly decouples inference latency from reasoning depth, a persistent challenge in current LLM design.

The experimental validation spans both controlled domains (cellular automata, graph retrieval) and practical applications (math reasoning), demonstrating that standard transformers and hybrid SSM-attention models fail on tasks where the sleep mechanism succeeds. Performance scaling with sleep duration indicates genuine reasoning improvement rather than superficial pattern matching.

For the AI industry, this work suggests a pathway beyond simple context windowing approaches. If further validated on large-scale models, sleep-like consolidation could enable longer effective context windows without proportional inference cost increases. This has profound implications for applications requiring sustained multi-step reasoning, from scientific problem-solving to complex planning tasks. The approach also highlights the value of borrowing mechanisms from neuroscience to solve engineering bottlenecks in deep learning.

• →Sleep-like consolidation converts recent context into persistent fast weights, enabling longer effective context without latency penalties
• →The mechanism offloads heavy computation to offline periods while preserving fast inference during active prediction
• →Performance improves measurably with increased sleep duration, particularly on tasks requiring deeper reasoning chains
• →Standard transformers and SSM-attention hybrids fail on benchmarks where the proposed method succeeds
• →The approach demonstrates practical value for math reasoning and retrieval tasks that demand multi-hop inference