简单的ReLU网络的费舍尔信息矩阵的近似光谱分解
Yoshinari Takeishi1, Masazumi Iida1, Jun'ichi Takeuchi1
1Faculty of Information Science and Electrical Engineering, Kyushu University, Motooka 744, Nishi-ku, Fukuoka, Fukuoka, 819-0395, Japan.
概括
本研究分析了ReLU网络的费舍尔信息矩阵 (FIM). 研究人员描述了它的固有值和固有向量,揭示了与重量矩阵和哈达马德产物相关的关键性质.
科学领域:
- 机器学习 机器学习
- 神经网络的神经网络的神经网络
- 信息理论 信息理论
背景情况:
- 费舍尔信息矩阵 (FIM) 对于理解统计模型中的参数灵敏度至关重要.
- 具有ReLU激活的神经网络被广泛使用,但它们的理论特性,像FIM一样,是复杂的.
- 分析深度学习模型的FIM可以提供对其学习动态和概括能力的见解.
研究的目的:
- 分析单个隐藏层神经网络的费舍尔信息矩阵 (FIM),使用修正线性单元 (ReLU) 激活函数.
- 描述FIM对输出层权重的自值和自向量.
- 建立FIM的理论特性,可以为网络设计和分析提供信息.
主要方法:
- 专注于重量向量的FIM (v),将隐藏层连接到标量输出.
- 在特定条件下导出FIM的理论性质.
- 使用数值计算来验证理论发现,特别是大量隐藏节点 (p ≈ 10000).
主要成果:
- 由于ReLU激活,FIM是非负的,第一个自值被确定为佩伦-弗罗贝尼乌斯自值.
- 对应于下一个最大自值的自空间被隐藏层重量矩阵 (W) 的行向量跨越.
- 第一个和第三个自值集群的联合自值空间由W的行向量的哈达马德乘积跨越.
结论:
- 该研究提供了FIM对ReLU网络的光谱属性的理论表征.
- 这些发现为FIM的结构和行为提供了洞察力,可能有助于理解网络优化和通用化.
- 数字验证支持理论结果,证实它们在许多隐藏单位的实际场景中的近似有效性.
更多相关视频
11:22Using Three-color Single-molecule FRET to Study the Correlation of Protein Interactions
Published on: January 30, 2018
10.1K
12:27Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations
Published on: February 15, 2017
7.0K
相关概念视频
01:11Residuals and Least-Squares Property
7.5K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
7.5K
01:20Region of Convergence of Laplace Tarnsform
614
The Region of Convergence (ROC) is a fundamental concept in signal processing and system analysis, particularly associated with the Laplace transform. The ROC represents an area in the complex plane where the Laplace transform of a given signal converges, determining the transform's applicability and utility.
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This...
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This...
614
01:19Routh-Hurwitz Criterion II
304
In the application of the Routh-Hurwitz criterion, two specific scenarios can arise that complicate stability analysis.
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
304
01:08Fisher's Exact Test
657
Fisher's exact test is a statistical significance test widely used to analyze 2x2 contingency tables, particularly in situations where sample sizes are small. Unlike the chi-squared test, which approximates P-values and assumes minimum expected frequencies of at least five in each cell, Fisher's exact test calculates the exact probability (P-value) of observing the data or more extreme results under the null hypothesis. This feature makes it especially valuable when the assumptions of...
657
01:13Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
99
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
99
01:17Region of Convergence
495
The z-transform is a powerful mathematical tool used in the analysis of discrete-time signals and systems. It is a crucial tool in the analysis of discrete-time systems, but its convergence is limited to specific values of the complex variable z. This range of values, known as the Region of Convergence (ROC), is fundamental in determining the behavior and stability of a system or signal. The ROC defines the region in the complex plane where the z-transform converges, which can take various...
495