Constraining Sequential Model Editing with Editing Anchor Compression

Researchers propose Editing Anchor Compression (EAC), a framework that addresses degradation of large language models' general abilities during sequential knowledge editing. By constraining parameter matrix deviations through selective anchor compression, EAC preserves over 70% of model performance while maintaining edited knowledge, advancing the practical viability of model editing as an alternative to expensive retraining.

The paper tackles a fundamental challenge in making large language models more maintainable and cost-effective. As LLMs become increasingly deployed in production systems, the need to update outdated or incorrect knowledge without full retraining becomes critical. Traditional model editing approaches have improved accuracy for specific factual updates but introduced an overlooked problem: successive edits progressively degrade the model's ability to perform unrelated tasks, reducing overall utility.

This degradation stems from parameter drift—the edited weight matrices deviate substantially from their original state as more edits accumulate. The deviation isn't gradual or uniform; it corrupts the knowledge associations the model learned during pretraining, effectively causing the model to "unlearn" capabilities outside the edited domains. This creates a practical bottleneck where organizations must choose between outdated knowledge and reduced general performance.

The EAC framework reframes editing as a compression problem rather than a replacement problem. By identifying editing anchors—parameters critical for encoding new relations while maintaining proximity to original weights—the approach constrains unwanted deviation. The methodology achieves meaningful results across multiple LLMs and editing techniques, suggesting it's broadly applicable rather than model-specific.

For AI developers and deployers, this research reduces the total cost of ownership for maintaining LLM systems. Instead of periodic full retraining cycles, organizations can perform incremental edits with preserved general capability. The 70% preservation threshold suggests EAC is production-ready for many use cases, particularly where occasional knowledge updates are needed without complete model replacement.

• →Sequential model editing causes parameter drift that degrades LLM general abilities on unrelated tasks.
• →Editing Anchor Compression constrains parameter deviation by selectively compressing editing information.
• →EAC preserves over 70% of general abilities while maintaining edited knowledge across multiple LLMs.
• →Framework proves effective across different editing methods and tasks, indicating broad applicability.
• →Research enables cost-effective knowledge maintenance as alternative to expensive full model retraining.