S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering

Researchers introduce S3MEM, a structured memory framework that improves how AI agents retrieve and answer questions about long trajectory histories. The system outperforms standard retrieval-augmented generation by organizing trajectories into scene-event units and using anchor-sensitive retrieval, achieving better accuracy with fewer tokens across multiple interactive environments.

S3MEM addresses a critical limitation in long-horizon AI agents: the inability to reliably answer questions about earlier events despite having extensive trajectory histories. The core innovation lies in reconceptualizing how agents store and retrieve information. Rather than treating trajectories as plain-text chunks indexed through generic retrieval, S3MEM structures memory into episodic units tied to scenes and events, enabling more precise evidence routing. This architectural shift proves particularly valuable for complex queries involving spatial relationships, temporal sequences, repeated events, and multi-hop reasoning.

The research reflects broader challenges in scaling AI agents to handle extended interactions. As agents accumulate longer histories, traditional RAG approaches struggle because they retrieve locally relevant fragments disconnected from the broader context chain necessary for accurate answers. S3MEM's anchor-sensitive retrieval mechanism actively seeks evidence aligned with query semantics rather than surface similarity, fundamentally changing how information flows from memory to inference.

The experimental validation spans diverse environments—Crafter, Jericho, SciWorld, and ALFWorld—demonstrating that S3MEM's advantages generalize beyond narrow use cases. The framework consistently outperforms vanilla RAG and achieves superior accuracy-efficiency frontiers compared to recent memory baselines, using dramatically fewer evidence tokens. This efficiency matters significantly for production deployments where computational costs scale with token usage.

Looking forward, this work validates the principle that memory interfaces deserve architectural consideration equivalent to model selection. As interactive AI agents become more prevalent in gaming, robotics, and other domains, structured episodic memory systems may become standard rather than optional. The research suggests future development should prioritize context-aware evidence routing and token-efficient retrieval mechanisms tailored to temporal and spatial reasoning.

• →S3MEM structures agent trajectories into scene-event episodic memory units rather than plain-text chunks, enabling more precise question answering.
• →Anchor-sensitive retrieval routes evidence based on query semantics, reducing chain-incomplete evidence problems in spatial, temporal, and multi-hop questions.
• →The framework achieves superior accuracy-efficiency frontiers while using dramatically fewer evidence tokens than competing approaches.
• →S3MEM consistently outperforms standard RAG and most recent baselines across four diverse interactive environments.
• →Results suggest structured memory interfaces provide stronger performance scaling than generic memory systems for long-horizon interactive AI.