分析深度卷积神经网络使用张量核和基于矩阵的透.
Kristoffer K Wickstrøm1, Sigurd Løkse1, Michael C Kampffmeyer1,2
1Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway.
Entropy (Basel, Switzerland)
|June 28, 2023
概括
本研究介绍了一种新的信息平面 (IP) 分析深度神经网络 (DNN),使用基于矩阵的Rényi.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 信息理论 信息理论
背景情况:
- 信息平面 (IP) 理论是分析深度神经网络 (DNN) 的强大工具,特别是它们的概括能力.
- 估计DNN和卷积神经网络 (CNN) 中高维层的相互信息 (MI) 对现有的IP方法构成重大挑战.
- 目前的知识产权分析技术是有限的,不能应用于大规模的CNN.
研究的目的:
- 提出一种新且可计算的方法,用于深层神经网络的信息平面分析,特别是大规模的CNN.
- 克服现有的MI估计器在处理高维度和卷积层方面的局限性.
- 为大规模神经网络的训练动态和概括能力提供新的见解.
主要方法:
- 开发了一种新的信息平面分析方法,利用基于矩阵的雷尼.
- 整合了带有Rényi的张量内核,以有效处理高维数据和卷积层.
- 杆化核心方法用于稳健的概率分布表示,独立于数据维度.
主要成果:
- 成功地应用了拟议的方法,对大型CNN进行了全面的IP分析.
- 研究了这些网络的不同训练阶段,揭示了它们行为的新方面.
- 提供了对大规模神经网络的训练动态的新见解,验证了对较小DNN的方法.
结论:
- 提出的基于矩阵的Rényi的和张量内核方法为深度神经网络的IP分析提供了可扩展和强大的解决方案.
- 这种方法可以研究复杂的,大规模的CNN,克服以前技术的局限性.
- 这些发现为了解DNN泛化和培训动态提供了新的视角.
相关概念视频
01:20Entropy and the Second Law of Thermodynamics
2.9K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
2.9K
02:39Entropy
30.4K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
30.4K
01:10Entropy Change in Reversible Processes
2.6K
In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
2.6K
01:24Convolution: Math, Graphics, and Discrete Signals
305
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
305
01:22Entropy within the Cell
10.8K
A living cell's primary tasks of obtaining, transforming, and using energy to do work may seem simple. However, the second law of thermodynamics explains why these tasks are harder than they appear. None of the energy transfers in the universe are completely efficient. In every energy transfer, some amount of energy is lost in a form that is unusable. In most cases, this form is heat energy. Thermodynamically, heat energy is defined as the energy transferred from one system to another that...
10.8K
01:17Convolution Properties II
239
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
239