When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory

Researchers present a scale-conditioned evaluation protocol for AI agent memory systems that tests whether stored evidence remains usable as irrelevant data accumulates. Testing across multiple memory architectures and language models reveals that reliability degrades unpredictably with scale, with some models exceeding computational budgets while others maintain performance, suggesting memory scalability claims must be conditioned on specific agent-interface-scale combinations.

This research addresses a critical gap in how AI agent memory systems are evaluated. Traditional benchmarks report static accuracy metrics that don't reflect real-world degradation as systems accumulate data over time. The scale-conditioned protocol introduces four diagnostic measures that track how memory retrieval reliability decays under evidence-preserving growth, where relevant task information stays fixed while irrelevant sessions increase—a scenario mimicking actual long-term system operation.

The findings expose surprising heterogeneity in memory system behavior. HippoRAG maintains computational efficiency but sacrifices 16-20 percentage points of reliability as scale increases. LiCoMemory's performance diverges dramatically across model sizes: Qwen3-8B violates resource constraints while larger variants remain efficient, suggesting memory interface design interacts unpredictably with model capacity. This heterogeneity challenges claims that memory architectures provide universal scalability improvements.

For the AI development community, these results force recalibration of how memory systems are marketed and deployed. A memory architecture cannot claim general scalability; it can only claim reliability within specific bounds defined by agent capability, interface design, retrieval budget, and scale range. This methodological contribution matters because production AI systems will operate at scales far exceeding current benchmarks, and failures under load could compromise critical applications.

The protocol establishes a framework for making empirically grounded scalability claims rather than aspirational ones. Future work will likely extend this evaluation approach across more architectures and identify failure mechanisms, potentially leading to memory designs that degrade gracefully rather than catastrophically.

• →Traditional memory benchmarks hide reliability degradation that emerges as irrelevant data accumulates at scale.
• →HippoRAG stays within computational budgets but loses 16-20% reliability; LiCoMemory's performance varies significantly across model sizes.
• →Memory system scalability is not universal—it depends jointly on agent architecture, interface design, budget constraints, and scale range.
• →The proposed evaluation protocol logs agent-memory trajectories and identifies usable-scale boundaries where performance drops below acceptable thresholds.
• →Different language model sizes interact differently with memory interfaces, suggesting no single memory design works optimally across all deployments.