Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

Researchers introduce HORMA, a hierarchical memory system for LLM agents that organizes experience into structured hierarchies with linked summaries and raw trajectories. The system achieves 22% token efficiency on long tasks while maintaining performance, addressing critical limitations in how language model agents manage working memory for multi-step reasoning.

HORMA represents a meaningful advance in solving a fundamental constraint of LLM-based agents: the stateless nature of large language models forces them to encode all relevant information in growing context windows, degrading reasoning quality and increasing computational costs. This research tackles the problem by proposing a two-stage memory architecture that mimics human organizational practices—structuring information hierarchically while maintaining access to underlying details. The distinction between failures caused by missing information versus misleading context is particularly valuable, as it allows the system to intelligently decide when to expand context rather than simply compressing it away.

The broader context involves an ongoing arms race to make language model agents practical for real-world tasks. As agents attempt longer horizons and more complex multi-step reasoning, memory management becomes the bottleneck. Previous approaches relied on lossy compression or embedding similarity, which fail to preserve the causal dependencies essential for sequential decision-making. HORMA's file-system-like hierarchy preserves both abstraction and detail simultaneously.

For developers building agent systems, this work offers concrete architectural improvements that could reduce deployment costs significantly. The reinforcement learning-trained navigation module represents an elegant solution to the problem of selecting minimal sufficient context, which directly translates to lower inference latency and costs. Testing across ALFWorld, LoCoMo, and LongMemEval demonstrates generalization beyond proprietary benchmarks.

Looking ahead, the key question involves whether such hierarchical memory approaches scale to vastly longer horizons and more diverse task domains. The success of HORMA suggests memory architecture—rather than raw model capacity—may be the limiting factor for practical long-horizon agents.

• →HORMA organizes agent memory hierarchically with summarized entities linked to raw trajectories, maintaining detail without token bloat
• →The system distinguishes between information-loss failures and context-overload failures to intelligently decide when to expand context
• →Achieves 22% token efficiency on long conversation tasks while maintaining or improving task performance
• →A trained RL-based navigation module selects minimal sufficient context by traversing the hierarchy, reducing latency on critical paths
• →Demonstrates consistent improvements across three diverse benchmarks, suggesting strong generalization to unseen tasks