PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Models

Researchers introduce PolarMem, a training-free memory framework that enhances vision-language models by explicitly tracking what has been verified as absent or excluded, not just what is similar. The system uses a polarized graph structure with positive and negative memory relations to reduce logical contradictions and improve reasoning reliability across multiple multimodal benchmarks.

PolarMem addresses a fundamental limitation in current vision-language model memory systems: their inability to remember what has been explicitly verified as absent or logically impossible. Traditional retrieval-augmented systems operate on positive association, fetching similar or previously observed information, but lack mechanisms to suppress contradictory or logically excluded possibilities. This gap creates vulnerability to hallucinations and logical inconsistencies in multimodal reasoning tasks.

The framework transforms frozen VLM signals into three memory states—HAS, NOT_HAS, and Uncertain—through semantic consistency verification and distributional partitioning. By storing these in a polarized graph with distinct positive and negative relations, PolarMem enables a logic-aware retrieval protocol that enforces logical consistency before semantic similarity during inference. This ordering prevents conflicting memories from contaminating the model context.

The technical innovation has meaningful implications for AI reliability. Testing across eight VLM backbones and six multimodal benchmarks demonstrates consistent improvements in retrieval-intensive tasks and measurable reductions in contradiction rates. For developers building production AI systems, this represents a practical path toward more reliable reasoning without requiring model retraining—a significant advantage for leveraging existing frozen models.

The broader significance lies in memory architecture design for AI systems. As multimodal models become embedded in critical applications, the ability to explicitly track negative evidence and logical constraints becomes increasingly important. This work demonstrates that negative memory relationships are not merely computational optimizations but essential components of robust reasoning systems. The open-source release enables rapid adoption and further research into how constraint-aware memory mechanisms can improve AI reliability.

• →PolarMem introduces explicit negative memory states (NOT_HAS) to prevent logical contradictions in vision-language model reasoning.
• →The training-free framework works with frozen VLM backbones, enabling compatibility with existing models without retraining.
• →Logic-aware retrieval prioritizes logical consistency over semantic similarity, reducing hallucination-prone retrievals.
• →Testing across six multimodal benchmarks shows consistent improvements in retrieval-intensive tasks and contradiction reduction.
• →Negative memory architecture represents a new paradigm for building more reliable multimodal AI systems.