Substrate Asymmetry in User-Side Memory: A Diagnostic Framework

Researchers reveal that large language model user-memory capabilities exhibit substrate asymmetry across three orthogonal dimensions—behavioral consistency, factual recall, and factual abstinence—with parametric methods (gamma-LoRA) excelling at style preservation while retrieval-augmented generation (RAG) excels at knowing when to abstain. The same neural circuits drive opposite-direction failures, and this tradeoff intensifies in heavily RLHF-tuned models, suggesting fundamental alignment costs to parametric personalization.

This arXiv paper addresses a critical but underexplored problem in LLM personalization: the assumption that user memory can be optimized as a single metric obscures competing failure modes. The research demonstrates that behavioral consistency and factual calibration operate on orthogonal axes, with parametric fine-tuning (gamma-LoRA) winning on style coherence while retrieval methods win on honest abstention when facts are absent. The causal analysis showing identical attention circuits load-bearing opposite effects reveals structural constraints rather than simple optimization failures.

The work's significance extends beyond academic taxonomy. As enterprises deploy personalized LLMs, this asymmetry becomes operationally critical—a user-adapted model that confidently hallcinates personalized falsehoods poses worse risks than one that maintains consistent voice. The alignment-tax finding on Llama-3.1 suggests that RLHF training amplifies this tradeoff rather than resolving it, indicating the problem is intrinsic to how parametric memory interacts with instruction-following objectives.

The practical implications shift substrate-selection from logit-based routing (calibration-aware) to question-classification (DistilBERT on query text alone). This counterintuitive finding suggests the solution space lies in smarter dispatch mechanisms rather than better parametric or retrieval methods alone. For developers building production personalization systems, this indicates hybrid architectures must choose routing mechanisms based on question semantics, not confidence scores. The diagnostic framework and mitigation strategies provide concrete pathways for practitioners, though the fundamental asymmetry likely persists across architectural choices.

• →User-memory substrates exhibit irreducible tradeoffs: parametric methods preserve behavioral style while retrieval methods better abstain from hallucinating absent facts.
• →The same neural circuits in attention layers 21-35 causally drive opposite effects, indicating structural rather than optimization-based constraints.
• →RLHF tuning intensifies substrate asymmetry rather than healing it, revealing an alignment cost to parametric personalization.
• →Substrate routing should use question-classification (DistilBERT) rather than logit-based calibration to select between parametric and retrieval approaches.
• →Instruction-following collapse, not substrate failure, explains gamma-LoRA underperformance on real data, fixable via eval-time logit masking.