注意力学习促进神经网络的泛化
IEEE transactions on neural networks and learning systems
|February 7, 2024
概括
研究人员通过分析与外分发 (OoD) 实例的训练来探索神经网络概括. 他们发现数据集-分心稳定性,测量对OoD数据的抵抗力,与概括性能有负相关性,这表明注意力学习可以改善结果.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 神经网络 (NN) 经常难以概括,特别是当在训练期间接触到分布外 (OoD) 数据时.
- 了解NN如何从噪音或无关数据中学习,对于提高其可靠性和性能至关重要.
研究的目的:
- 调查 OoD 例子对神经网络概括的影响.
- 引入和量化一个新的指标,数据集-分心稳定性,以衡量网络对无关紧要的培训数据的弹性.
- 建立注意力学习,数据集-分心稳定性和概括界限之间的理论联系.
主要方法:
- 检查了神经网络行为,当训练在数据集和没有OOD例子.
- 在各种 NN 架构 (VGG,ResNet,WideResNet,ViT) 和优化器中提出并测量了数据集分散稳定性.
- 利用内在维度 (ID) 来将复杂分布的学习过程分解为更简单的.
主要成果:
- 在广泛的CIFAR-10/100实验中,观察到数据集分散稳定性和概括性之间的负相关性.
- 具有更高分心稳定的网络表现出更好的泛化.
- 通过分解基于分布简单性的学习过程来获得更紧密的概括界限.
结论:
- 专注学习,以OOD示例的影响较低为特征,是深度学习中实现强烈概括的关键.
- 数据集-分心稳定性提供了一种可量化的衡量方法,以理解和潜在地改善概括.
- 这些发现为设计新的算法铺平了道路,这些算法促进了注意力学习并增强了模型的稳定性.
相关概念视频
Generalization, Discrimination, and Extinction
559
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
559
Higher Mental Functions of Brain: Learning and Memory
809
Memory is one of the most vital higher mental functions of the brain. Memory is closely related to learning because it enables us to retain information and experiences from our past to use them in our present life. It also helps us to remember facts, events, and skills, such as riding a bike or swimming. There are two types of memory — declarative memory, which involves memorizing facts or events, and procedural memory, which enables us to remember how to do something like writing or...
809
Neuroplasticity
361
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
361
Neural Circuits
1.2K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.2K
Associative Learning
370
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
370
Long-term Potentiation
55.2K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
55.2K


