密集的关联记忆的指数容量
Carlo Lucibello1, Marc Mézard1
1Department of Computing Sciences, Bocconi University, Milano 20136, Italy and Bocconi Institute for Data Science and Analytics (BIDSA), Milano 20136, Italy.
Physical review letters
|March 1, 2024
概括
一般化的霍普菲尔德网络存储了指数级的许多模式,与深度学习注意力机制联系在一起. 统计力学分析揭示了不同模式类型的存储容量值和不同的相位图.
科学领域:
- 统计力学就是统计力学.
- 机器学习 机器学习
- 神经网络的神经网络的神经网络
背景情况:
- 一般化的霍普菲尔德网络提供指数级存储容量 (P=exp(αN)).
- 这些网络显示出与变压器架构中的注意力机制的联系.
- 了解存储限制和检索属性对于先进的AI模型至关重要.
研究的目的:
- 在一般化霍普菲尔德网络中分析一种通用模式组合家族.
- 为了确定准确的模式检索 (α1) 的非对称值.
- 为了确定最大储存负载 (αc) 的下限,并分析吸引盆地大小.
主要方法:
- 使用统计力学分析进行准确的非对称计算.
- 调查通用模式合集. 调查通用模式合集.
- 关于高斯式和球形模式的详细案例研究.
主要成果:
- 为典型的模式检索 (α1) 推导出精确的非对称值.
- 为最大储存负载 (αc) 设定了下限.
- 具有特色的吸引力盆地尺寸.
- 揭示了高斯式和球形模式的丰富且质量不同的相位图.
结论:
- 一般化的霍普菲尔德网络具有与深度学习相关的显著存储能力.
- 统计力学为网络性能限制提供了精确的洞察力.
- 模式组合的选择极大地影响网络行为和相位图.
相关概念视频
Higher Mental Functions of Brain: Learning and Memory
773
Memory is one of the most vital higher mental functions of the brain. Memory is closely related to learning because it enables us to retain information and experiences from our past to use them in our present life. It also helps us to remember facts, events, and skills, such as riding a bike or swimming. There are two types of memory — declarative memory, which involves memorizing facts or events, and procedural memory, which enables us to remember how to do something like writing or...
773
Storage
85
A schema is a mental framework that helps individuals organize and interpret information. Schemata, formed from previous experiences, influence how we process new information: how we encode it, the inferences we make, and how we retrieve it. For instance, a schema for what a typical classroom looks like might include desks, a teacher's desk, a whiteboard, and students in such an environment. This expectation helps us quickly understand and navigate new classrooms without needing to analyze...
85
Long-Term Memory
159
Long-term memory is a relatively permanent type of memory, capable of storing vast amounts of information over extended periods. Its storage capacity is generally considered unlimited.
Long-term memory can be categorized into two primary types: explicit and implicit memory. Explicit memory, also known as declarative memory, involves the conscious recollection of information that we deliberately try to remember, recall, and articulate. This type of memory encompasses specific facts, events, and...
Long-term memory can be categorized into two primary types: explicit and implicit memory. Explicit memory, also known as declarative memory, involves the conscious recollection of information that we deliberately try to remember, recall, and articulate. This type of memory encompasses specific facts, events, and...
159
Explicit Memories
137
Explicit memories, also known as declarative memories, are consciously remembered, recalled, and reported. Studying for a chemistry exam involves material that will become part of explicit memory. There are two types of explicit memory: episodic and semantic.
Episodic memory contains information about personally experienced events and is reported as a story. An example of episodic memory is recalling a birthday celebration. This type of memory includes the what, where, and when of an event, as...
Episodic memory contains information about personally experienced events and is reported as a story. An example of episodic memory is recalling a birthday celebration. This type of memory includes the what, where, and when of an event, as...
137
Chunking
99
Chunking is a powerful cognitive technique that improves short-term memory retention by organizing information into smaller, more manageable units. The brain, limited by working memory capacity, can more easily process and store information when it is divided into "chunks" rather than presented as discrete, unrelated elements. Chunking is especially useful when dealing with large amounts of information, such as numerical sequences, words, or complex ideas.
The principle behind chunking...
The principle behind chunking...
99
Interference and Decay
140
Forgetting is a complex cognitive phenomenon influenced by several factors, among which interference and decay are particularly prominent. These processes explain why individuals often struggle to retrieve specific information from memory, leading to lapses in recall that can be observed in everyday situations.
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
140


