噪音和信息之间的界限用于过神经网络重量矩阵
Max Staats1, Matthias Thamm1, Bernd Rosenow1
1Institut für Theoretische Physik, Universität Leipzig, Brüderstrasse 16, 04103 Leipzig, Germany.
Physical review. E
|September 19, 2023
概括
过度参数化的深度神经网络 (DNN) 显示了随机和学习信息之间的频谱边界. 一个新的噪声过算法通过删除和减少单数值来改善DNN概括,特别是当使用噪声标签进行训练时.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 数据科学数据科学数据科学
背景情况:
- 深度神经网络 (DNN) 经常表现出过度参数化,导致具有部分随机特征的权重矩阵.
- 这些矩阵的奇数值谱显示了随机组件和学习信息之间的边界,与波特-托马斯分布相比较.
研究的目的:
- 为深度神经网络引入一种新的噪声过算法.
- 为了减轻DNN重量矩阵的奇数值谱内的随机噪声的影响.
- 提高DNN的通用化性能,特别是当使用噪音标签进行训练时.
主要方法:
- 在过度参数化的深度神经网络中分析单一值谱.
- 将奇点向量与波特-托马斯分布进行比较,以确定随机性-信息边界.
- 开发一种噪声过算法,可以删除小单数值并减少大单数值.
主要成果:
- 拟议的噪声过算法有效地抵消了单一值谱中的噪声和信息之间的水平排斥.
- 在应用过算法后,在经过标签噪声训练的网络中观察到普遍化性能的显著改善.
结论:
- 噪声过是一种可行的策略,通过解决重量矩阵的光谱特性来增强DNN概括.
- 该算法的有效性在涉及杂训练数据的场景中特别明显,突出其实际实用性.
相关概念视频
Neural Regulation
39.5K
Digestion begins with a cephalic phase that prepares the digestive system to receive food. When our brain processes visual or olfactory information about food, it triggers impulses in the cranial nerves innervating the salivary glands and stomach to prepare for food.
39.5K
Difference from Background: Limit of Detection
6.4K
The limit of detection (LOD) is the smallest amount of analyte that can be distinguished from the background noise. The LOD value corresponds to the concentration at which the analyte signal is three times larger than the standard deviation of the blank signal. Below this value, the analyte signal cannot be differentiated from the background noise. It is calculated by dividing the calibration slope by 3 times the standard deviation of the blank signals.
The LOD indicates the presence or absence...
The LOD indicates the presence or absence...
6.4K
Active Filters
855
Active filters are electronic circuits that use operational amplifiers (op-amps), resistors, and capacitors to filter out unwanted frequency components from a signal. A first-order low-pass active filter is designed to pass signals with a frequency lower than a certain cutoff frequency and attenuate frequencies higher than that cutoff frequency. The transfer function for a first-order low-pass active filter is:
855
Weighted Mean
5.2K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
5.2K
Classification of Signals
519
In signal processing, signals are classified based on various characteristics: continuous-time versus discrete-time, periodic versus aperiodic, analog versus digital, and causal versus noncausal. Each category highlights distinct properties crucial for understanding and manipulating signals.
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
519
Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K


