ASMem: Anchor sparse memory for multi-domain knowledge editing of large language models.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) face challenges with outdated or inconsistent knowledge across multiple domains.
• Maintaining reliability, generality, and locality during knowledge editing is crucial for LLM performance.
• Catastrophic forgetting remains a significant issue in editing LLMs.

Purpose of the Study:

• To investigate the trade-offs among reliability, generality, and locality in multi-domain knowledge editing.
• To propose a novel method, anchor sparse memory (ASMem), to address knowledge editing challenges in LLMs.
• To evaluate ASMem's effectiveness in improving LLM consistency and accuracy across diverse knowledge domains.

Main Methods:

• ASMem is a plug-and-play module isolating edits into parallel memory-specific parameters.
• A novel anchor prototype routing mechanism ensures precise query-to-memory alignment.
• Unsupervised clustering partitions memory modules into distinct semantic domains for efficient knowledge management.

Main Results:

• ASMem significantly outperforms existing methods in reliability and generalization for multi-domain knowledge editing.
• The method maintains high locality, preserving unrelated knowledge effectively.
• ASMem demonstrates robust performance across different model families (LLaMA, Qwen) and scales (0.5B to 8B), in both English and Chinese.

Conclusions:

• ASMem offers an effective solution for updating and maintaining knowledge in large language models across multiple domains.
• The proposed method mitigates catastrophic forgetting and improves overall LLM performance and consistency.
• ASMem shows broad applicability and generalization capabilities for future LLM development.