Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

Researchers introduce Memory-Efficient Looped Transformer (MELT), an architecture that decouples reasoning depth from memory consumption in recurrent language models. MELT replaces the standard approach of maintaining separate Key-Value caches per reasoning loop with a single shared cache per layer, updated via learnable gating, achieving constant-memory iterative reasoning comparable to standard LLMs while outperforming them on benchmarks.

MELT addresses a critical scalability bottleneck in recurrent language models like Ouro, which perform multi-step reasoning through iterative computation. The fundamental problem with existing architectures is that memory consumption grows linearly with reasoning depth as each iteration adds its own Key-Value cache, making deep reasoning computationally prohibitive. This limitation directly constrains how much computational work these models can perform internally without generating intermediate tokens.

The innovation lies in MELT's shared cache mechanism coupled with learnable gating, which allows reasoning iterations to update and refine representations without multiplicative memory overhead. The researchers employ a two-phase training strategy—interpolated transition followed by attention-aligned distillation—to ensure stable learning under this novel constraint. This approach builds on established techniques from the LoopLM framework while fundamentally restructuring how information persists across reasoning steps.

For the broader AI infrastructure ecosystem, this work demonstrates that memory efficiency and reasoning capability need not trade off against each other. Models fine-tuned from Ouro achieve superior performance relative to comparable standard LLMs while maintaining standard memory footprints, suggesting that architectural innovations can unlock reasoning capabilities without requiring proportional hardware investment. This matters because reasoning performance has become increasingly important for competitive language models, yet the computational cost remains a barrier for broader deployment.

The practical implications extend to production environments where memory constraints limit model capacity or iteration depth. Developers can now potentially achieve longer reasoning chains with existing hardware. Future work likely explores scaling these constant-memory reasoning approaches to larger models and more complex reasoning tasks.

• →MELT decouples reasoning depth from memory consumption through a single shared Key-Value cache per layer updated via learnable gating
• →Models fine-tuned from Ouro parameters using MELT achieve better performance than standard LLMs of comparable size with dramatically reduced memory usage
• →The two-phase training procedure (interpolated transition and attention-aligned distillation) enables stable learning without the memory scaling issues of prior recurrent architectures
• →Constant-memory iterative reasoning becomes feasible through architectural innovation rather than just raw computational scaling
• →This approach addresses a critical bottleneck in deploying reasoning-capable models where memory overhead previously limited practical scalability