大型语言模型的马赛克记忆
Igor Shilov1, Matthieu Meeus1, Yves-Alexandre de Montjoye2
1Department of Computing and Data Science Institute, Imperial College London, London, UK.
Nature communications
|January 29, 2026
概括
大型语言模型 (LLM) 通过组装类似的序列来记住数据,而不仅仅是精确的重复,这个过程被称为马赛克记忆. 这种语法记忆影响了隐私和模型评估.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 越来越普遍.
- 了解LLM对培训数据的记忆是至关重要的.
- 目前的假设将记忆仅与数据重复联系在一起.
研究的目的:
- 研究LLMs的记忆机制.
- 引入和定义"马赛克记忆"的概念.
- 分析LLM记忆对数据隐私和模型评估的影响.
主要方法:
- 在主要的法学士课程中研究记忆.
- 量化了模糊重复对记忆的贡献.
- 分析了记忆的语法与语义性质.
- 检查了现实数据中模糊重复的普遍性.
主要成果:
- 这种LLM表现出"马赛克记忆",从相似的信息中汇集信息,而不仅仅是相同的序列.
- 模糊的重复物对记忆有很大的贡献,与精确的重复物相比.
- 记忆主要是语法,而不是语义,尽管LLM的推理能力.
- 无处不在的模糊重复存在于现实世界的数据中,避免了脱重复.
结论:
- 士学位记忆是一个复杂的马赛克过程,不仅仅是基于精确的重复.
- 马赛克内存对数据隐私,保密性和模型实用性有重大影响.
- 目前的脱重复技术不足以解决训练数据中的模糊重复.
- 需要进一步的研究来理解和减轻与马赛克记忆相关的风险.
相关概念视频
The Fluid Mosaic Model
177.9K
The fluid mosaic model was first proposed as a visual representation of research observations. The model comprises the composition and dynamics of membranes and serves as a foundation for future membrane-related studies. The model depicts the structure of the plasma membrane with a variety of components, which include phospholipids, proteins, and carbohydrates. These integral molecules are loosely bound, defining the cell’s border and providing fluidity for optimal function.
177.9K
Fluid Mosaic Model
16.2K
Scientists identified the plasma membrane in the 1890s and its principal chemical components (lipids and proteins) by 1915. The model for plasma membrane structure, proposed in 1935 by Hugh Davson and James Danielli, was the first model to be widely accepted in the scientific community. The model was based on the plasma membrane's "railroad track" appearance in early electron micrographs. Davson and Danielli theorized that the plasma membrane's structure resembled a sandwich...
16.2K
Language
909
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
909
Components of Language
817
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
817
Language Development
911
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
911
Language and Cognition
797
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
797


