HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

Researchers present HoReN, a novel method for editing large language models that preserves original knowledge while incorporating new information through a codebook-based external memory system. The approach uses Hopfield networks and angular similarity retrieval to handle up to 50,000 sequential edits, significantly outperforming existing model editing techniques that degrade at scale.

HoReN addresses a fundamental challenge in maintaining deployed large language models: updating factual knowledge without destabilizing the underlying model. Current approaches face a critical tradeoff. Direct weight modification methods progressively corrupt preserved knowledge even with protective constraints, while external memory approaches struggle with routing efficiency and performance degradation as edit volume increases. The proposed solution elegantly combines discrete codebook storage with modern Hopfield networks, treating each codebook entry as both a knowledge memory key and a stored pattern. This dual interpretation enables sophisticated retrieval mechanics.

The technical innovation centers on three mechanisms that work synergistically. By projecting both keys and queries onto the unit hypersphere, the system uses angular similarity rather than magnitude-based matching, eliminating a common failure mode where semantically identical prompts with different phrasings fail to retrieve the correct edit. The damped Hopfield attractor dynamics further refine queries, allowing paraphrased versions to converge toward the correct stored pattern while leaving unrelated queries undisturbed. This prevents off-target edits from corrupting unrelated knowledge.

The experimental results demonstrate substantial practical advantages. HoReN maintains performance above 0.9 across 50,000 sequential edits on standard benchmarks, whereas prior editors collapse or degrade severely before reaching 10,000 edits. This scalability directly translates to production value for organizations managing evolving language model deployments in dynamic information environments. The approach applies broadly across diverse evaluation datasets, suggesting robust generalization. For practitioners deploying large language models in applications requiring frequent factual updates—knowledge bases, news systems, and real-time information retrieval—this represents a meaningful advancement in model maintainability without expensive retraining cycles.

• →HoReN enables stable editing of large language models across 50,000+ sequential updates while maintaining performance above 0.9
• →Hypersphere projection and angular similarity matching eliminate magnitude-driven mismatches between prompts and their rephrasings
• →Hopfield attractor dynamics allow paraphrased queries to converge to correct patterns while protecting unrelated knowledge
• →The codebook-based approach preserves base model weights, avoiding the knowledge corruption that accumulates with direct parameter modification
• →Performance scales significantly beyond prior methods, which typically collapse or severely degrade before 10,000 edits