Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory

Researchers introduce BudgetMem, a runtime memory framework for LLM agents that uses query-aware routing to dynamically allocate computational resources across memory modules at three cost tiers. The system employs reinforcement learning to optimize the performance-cost trade-off, demonstrating improvements over static memory approaches across multiple benchmark datasets.

BudgetMem addresses a fundamental constraint in deploying LLM agents at scale: memory management beyond single context windows. Traditional approaches construct memory offline without considering specific query requirements, leading to inefficient resource allocation and potential loss of task-critical information. This research bridges that gap by introducing a runtime routing system that adapts memory processing based on actual query needs.

The framework's three-tier architecture (Low/Mid/High budget) represents a pragmatic approach to production deployment where computational cost directly impacts operational expenses. By training a lightweight neural policy through reinforcement learning, BudgetMem enables explicit control over accuracy-cost frontiers without requiring manual tuning for each use case. This addresses a longstanding challenge in AI systems: balancing model capability against infrastructure costs in real-world applications.

The tiering strategies—implementation complexity, reasoning behavior, and module model size—reveal domain-specific trade-offs that practitioners can leverage based on their constraints. Performance improvements on LoCoMo, LongMemEval, and HotpotQA benchmarks suggest practical applicability beyond theoretical domains. For organizations operating LLM agents, this means potential cost reductions without sacrificing quality on high-priority tasks.

The research contributes to the broader trend of making AI systems more efficient through intelligent resource allocation. Rather than scaling all components equally, BudgetMem selectively allocates resources where they matter most, reducing the total computational footprint of agent systems. Future work likely explores how such adaptive routing generalizes to other components of complex AI systems, potentially setting a pattern for efficiency-focused architecture in next-generation AI infrastructure.

• →BudgetMem enables runtime, query-aware memory routing that dynamically adjusts computational cost based on task requirements.
• →Three complementary tiering strategies (implementation, reasoning, capacity) offer different accuracy-cost trade-offs under varying budget constraints.
• →Reinforcement learning-trained router provides explicit performance-cost control without manual configuration for specific use cases.
• →Framework demonstrates measurable improvements on benchmark datasets while reducing memory construction overhead in high-budget settings.
• →Adaptive memory allocation approach suggests a scalable pattern for efficient deployment of LLM agents in production environments.