深度神经网络有一个内置的奥卡姆剃须刀
Chris Mingard1,2, Henry Rees1, Guillermo Valle-Pérez1
1Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford, UK.
Nature communications
|January 14, 2025
概括
深度神经网络 (DNN) 的成功是由于架构,训练和数据结构的融合. 结构化数据和对简单函数的偏见是DNN性能的关键.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 计算神经科学是一种神经科学.
背景情况:
- 过度参数化的深度神经网络 (DNN) 显示出了显著的性能.
- 这种成功的原因归因于网络架构,训练算法和数据结构之间的相互作用.
- 解开这些组成部分对于理解监督学习至关重要.
研究的目的:
- 研究深度神经网络 (DNN) 中的网络架构,训练算法和数据结构之间的相互作用.
- 应用贝叶斯框架来分析DNN表达的函数.
- 了解导致DNN在监督学习中的成功的诱导偏差.
主要方法:
- 用贝叶斯的方法来建模DNN所代表的函数.
- 网络架构通过探索有序和混乱制度之间的过渡来变化.
- 使用布尔函数分类数据上的函数误差谱来近似估计概率.
- 深度神经网络使用随机梯度下降进行训练.
主要成果:
- 贝叶斯框架准确地预测了函数的后部分布.
- 该研究表明,在经过随机梯度下降训练的DNN测量后部的准确预测.
- 分析显示,结构化数据起着至关重要的作用.
结论:
- DNN的成功归因于结构化数据和奥卡姆的剃须刀般的诱导偏见.
- 这种偏见有利于科尔摩戈罗夫的简单函数,抵消函数复杂性的指数增长.
- 这些发现强调了数据结构和DNN中固有的偏差对于有效的监督学习的重要性.
相关概念视频
Neural Circuits
1.0K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.0K
Association Areas of the Cortex
5.0K
Association areas are regions of the cerebral cortex that do not have a specific sensory or motor function. Instead, they integrate and interpret information from various sources to enable higher cognitive processes such as memory, learning, and decision-making. Some key association areas include the following:
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
5.0K
Parallel Processing
145
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
145
Reducing Line Loss
143
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
143
Anatomy of the Eyeball
5.9K
The eye is a spherical, hollow structure composed of three tissue layers. The outer layer — the fibrous tunic, comprises the sclera — a white structure — and the cornea, which is transparent. The sclera encompasses some of the ocular surface, most of which is not visible. However, the 'white of the eye' is distinctively visible in humans compared to other species. The cornea, a clear covering at the front of the eye, enables light penetration. The eye's middle...
5.9K
Associative Learning
286
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
286


