线性反向传播及其收的理论观点
IEEE transactions on pattern analysis and machine intelligence
|January 15, 2024
概括
与标准的反向传播 (BP) 相比,线性反向传播 (LinBP) 为深度神经网络训练和对抗性攻击提供了更快的融合. 本研究为LinBP提供了理论分析和实验验证.
科学领域:
- 深度学习是一种深度学习.
- 机器学习理论机器学习理论
- 神经网络优化神经网络优化
背景情况:
- 逆向传播 (BP) 是深度神经网络 (DNN) 中梯度计算的标准算法,对于训练和对抗应用至关重要.
- 一种线性变体,LinBP,在生成黑子攻击的可转移对抗示例方面取得了经验上的成功.
- 对于LinBP的理论分析和收性质仍未得到充分研究.
研究的目的:
- 为线性反向传播 (LinBP) 算法提供理论分析.
- 研究LinBP在深度神经网络训练和对抗性攻击场景中的融合特性.
- 为了补充和扩展之前关于LinBP的经验发现.
主要方法:
- 在神经网络学习任务中对LinBP的融合进行理论分析.
- 在相同的超参数设置下对LinBP与标准反向传播 (BP) 的比较研究.
- 进行广泛的实验验证,以证实理论结果.
主要成果:
- 与标准BP相比,LinBP在对抗性攻击生成和模型训练方面表现出更快的趋同.
- 理论分析支持观察到的LinBP.的更快的趋同.
- 实验结果在各种任务中验证了理论发现.
结论:
- 在特定的深度学习应用中,LinBP为BP提供了一个潜在的更有效的替代方案.
- 建立了LinBP更快收的理论基础.
- 对LinBP的进一步研究可能会导致对抗性强度和训练效率的提高.
相关概念视频
01:26Linear Approximation in Frequency Domain
91
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
91
01:21Linear Approximation in Time Domain
83
Nonlinear systems often require sophisticated approaches for accurate modeling and analysis, with state-space representation being particularly effective. This method is especially useful for systems where variables and parameters vary with time or operating conditions, such as in a simple pendulum or a translational mechanical system with nonlinear springs.
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
83
01:17Linear Circuits
406
A linear circuit is characterized by its output having a direct proportionality to its input, adhering to the linearity property, which encompasses the principles of homogeneity (scaling) and additivity. Homogeneity dictates that when the input, also referred to as the excitation, is multiplied by a constant factor, the output, known as the response, is correspondingly scaled by the same constant factor. For instance, if the current is multiplied by a constant 'k,' the voltage likewise...
406
01:26Classification of Systems-I
188
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
188
01:23Linear time-invariant Systems
262
A system is linear if it displays the characteristics of homogeneity and additivity, together termed the superposition property. This principle is fundamental in all linear systems. Linear time-invariant (LTI) systems include systems with linear elements and constant parameters.
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
262
01:20Region of Convergence of Laplace Tarnsform
555
The Region of Convergence (ROC) is a fundamental concept in signal processing and system analysis, particularly associated with the Laplace transform. The ROC represents an area in the complex plane where the Laplace transform of a given signal converges, determining the transform's applicability and utility.
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This...
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This...
555