MEMENTO: Teaching LLMs to Manage Their Own Context

Researchers introduce MEMENTO, a method enabling large language models to compress their reasoning into dense summaries (mementos) organized into blocks, reducing KV cache usage by 2.5x and improving throughput by 1.75x while maintaining accuracy. The technique is validated across multiple model families using OpenMementos, a new dataset of 228K annotated reasoning traces.

MEMENTO addresses a fundamental inefficiency in how reasoning models operate: they generate long, unstructured chains of thought without compressing intermediate states, leading to massive context windows and computational overhead. By teaching models to autonomously segment reasoning into blocks and create dense mementos—textual summaries of key information—the method enables forward reasoning using only these compressed states rather than full context histories.

This research builds on growing recognition within the AI community that efficient reasoning requires better state management. Previous approaches relied on external compression or fixed summarization strategies, whereas MEMENTO enables models to learn compression patterns through supervised fine-tuning on the newly released OpenMementos dataset. The approach generalizes across different architectures (Qwen3, Phi-4, Olmo) and scales (8B to 32B parameters), suggesting fundamental utility rather than model-specific optimization.

The practical implications are substantial. A 2.5x reduction in peak KV cache directly decreases memory requirements and latency, while 1.75x throughput improvements enhance inference efficiency at scale. This matters for both cloud providers running inference services and developers deploying reasoning models on constrained hardware. The discovery of a dual information stream—where both memento text and implicit KV states carry information—indicates models develop sophisticated compression strategies that deserve further investigation.

Future work should examine whether MEMENTO principles apply to longer reasoning chains (100+ steps) and non-mathematical domains. Integration with reinforcement learning for accuracy improvement suggests the method is compatible with advanced training techniques rather than merely a post-hoc compression approach.

• →MEMENTO reduces KV cache requirements by 2.5x while maintaining reasoning accuracy across math, science, and coding tasks
• →The method teaches models to autonomously compress reasoning into dense memento summaries through supervised fine-tuning
• →OpenMementos dataset of 228K annotated traces enables generalization across multiple LLM architectures and parameter scales
• →vLLM integration achieves 1.75x throughput improvement, making efficient reasoning deployment more practical
• →Dual information streams in mementos reveal models store both explicit and implicit compression that jointly preserves reasoning quality