Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning

Researchers introduce TASM (Task-Aware Structured Memory), a training-free framework that optimizes how multi-modal large language models compress and retrieve information during in-context learning. The method addresses critical scalability limitations by using task-aware compression, structure-preserving token merging, and dynamic memory hierarchies to maintain performance while reducing computational costs.

TASM represents a meaningful advance in addressing fundamental constraints that limit multi-modal LLM deployment at scale. Current MLLMs struggle with context window limitations and exponentially growing key-value cache costs when processing long visual and textual sequences—a bottleneck that directly impacts real-world applications from document analysis to video understanding. The framework tackles this through three complementary mechanisms: task-vector guided compression replaces arbitrary token removal with learned task relevance, bipartite graph matching preserves semantic structure during token aggregation, and hierarchical memory organization enables query-adaptive retrieval rather than static compression.

The approach emerges from growing recognition that naive compression strategies degrade performance by destroying the underlying manifold structure of learned representations. Previous methods either rigidly prune tokens or assign importance scores based on individual samples, both introducing distortions that cascade through downstream inference. TASM's training-free design is practically significant—it integrates with existing models without retraining costs.

For developers and organizations deploying MLLMs, this directly addresses operational efficiency. Reduced KV cache requirements translate to lower memory footprints, faster inference latency, and decreased computational costs—particularly valuable for edge deployments and resource-constrained environments. The dynamic retrieval capability allows single compressed memories to serve multiple query types effectively, improving utilization compared to static compression.

Future validation will focus on evaluating TASM across diverse multi-modal tasks and sequence lengths. The framework's interaction with emerging efficient architectures and whether task-awareness generalizes across significantly different domains will determine adoption breadth.

• →TASM uses task-vector guided compression to replace sample-specific pruning with task-level relevance signals across demonstrations
• →Bipartite graph matching preserves semantic structure during token aggregation, avoiding destructive information loss
• →Hierarchical Core Memory and Latent Bank design enables query-adaptive dynamic retrieval instead of static compression
• →Training-free framework integrates with existing MLLMs without retraining requirements
• →Addresses critical scalability bottleneck of KV cache costs in long multi-modal sequences