MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents

MemRouter is a new memory management system for conversational AI agents that uses lightweight embedding-based routing instead of expensive LLM generation to decide which conversation turns to store. The approach achieves 52.0 F1 score versus 45.6 for LLM-based alternatives while reducing latency from 970ms to 58ms, suggesting memory admission can be effectively learned through supervised classification rather than generative models.

MemRouter addresses a fundamental efficiency problem in long-term conversational AI systems. As conversational agents maintain extended interactions, they must decide what information deserves storage in external memory—a task that current systems handle through expensive autoregressive LLM generation at every conversational turn. This approach creates computational bottlenecks that limit scalability and responsiveness.

The innovation decouples memory management from answer generation, replacing per-turn LLM decoding with a specialized embedding-based router trained on only 12 million parameters. The system encodes conversation turns with context, projects them through a frozen LLM backbone, and uses lightweight classification heads to predict storage decisions. This architectural separation reflects a broader trend in AI toward modular, task-specific components rather than monolithic language models handling all functions.

The performance improvements are substantial. MemRouter achieves 52.0 F1 compared to 45.6 for LLM-based memory managers across all question categories, with non-overlapping confidence intervals confirming statistical significance. Latency reduction from 970ms to 58ms represents a 16x speedup, critical for production conversational systems where user experience depends on response times. The factorial analysis reveals that learned admission policies contribute +10.3 F1 improvement over random storage, while category-specific prompting adds +5.2 points.

For developers building conversational AI applications, MemRouter demonstrates that specialized routers can outperform general-purpose language models at specific tasks. This approach enables faster, more scalable systems without sacrificing answer quality, as the QA backbone remains unchanged. The work validates a design philosophy where different components optimize for their specific functions rather than asking a single model to excel at everything.

• →MemRouter achieves 52.0 F1 on memory routing versus 45.6 for LLM-based alternatives using only 12M trainable parameters
• →Embedding-based routing reduces memory-management latency from 970ms to 58ms, enabling faster conversational interactions
• →Learned admission policies improve performance by +10.3 F1 over random storage and +5.2 over generic prompting
• →The approach decouples memory management from answer generation, allowing each component to optimize independently
• →Results suggest specialized routers outperform general-purpose LLMs at specific memory-management tasks in conversational AI