基于RMSProp的隐式规范化在深层低级别矩阵因数分解中的动力学理论
概括
本研究引入了景观分析,以解释使用RMSProp优化在深度网络中的隐性规范化. 它显示RMSProp辅助点逃脱,导致对矩阵重建任务的更快收.
科学领域:
- 机器学习 机器学习
- 深度学习理论 深度学习理论
- 优化算法 优化算法
背景情况:
- 从渐变优化的隐式规范化有助于神经网络的泛化.
- 现有的理论分析深度矩阵分解 (DMF) 和离散梯度动力学.
- 离散梯度动力学是RMSProp等自适应梯度方法的特征,但对于深度网络来说是复杂的.
研究的目的:
- 从理论和实验上解释基于RMSProp的深度网络中的隐性规范化.
- 引入景观分析,重点关注位和当地最小值.
- 调查学习率对点逃跑 (SPE) 的影响.
主要方法:
- 使用景观分析开发了一种离散梯度动态方法.
- 在深度网络中分析了点逃逸 (SPE) 动态.
- 在使用DMF和SPE的Rank-R矩阵重建中证明了收性质.
主要成果:
- 证明 DMF 在 SPE 的 R 阶段之后汇聚到第二阶段的临界点,用于 R 级矩阵重建.
- 分析了在SPE期间逃离高原所需的时间.
- 实验验证了低等级矩阵,图像和汉克尔矩阵重建的发现.
结论:
- RMSProp比梯度下降 (GD) 和AdaGrad表现出更强的隐性规范化,但比Adam.
- 景观分析为了解深度网络中的隐性规范化提供了一种可行的方法.
- 理论框架适用于GD和Adam,但不适用于AdaGrad.
更多相关视频
11:11Longitudinal Measurement of Extracellular Matrix Rigidity in 3D Tumor Models Using Particle-tracking Microrheology
Published on: June 10, 2014
11.5K
10:44Inherent Dynamics Visualizer, an Interactive Application for Evaluating and Visualizing Outputs from a Gene Regulatory Network Inference Pipeline
Published on: December 7, 2021
2.1K
相关概念视频
Residuals and Least-Squares Property
7.3K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
7.3K
Relation between Poisson's ratio, Modulus of Elasticity and Modulus of Rigidity
253
Deformation occurs in axial and transverse directions when an axial load is applied to a slender bar. This deformation impacts the cubic element within the bar, transforming it into either a rectangular parallelepiped or a rhombus, contingent on its orientation. This transformation process induces shearing strain. Axial loading elicits both shearing and normal strains. Applying an axial load instigates equal normal and shearing stresses on elements oriented at a 45° angle to the load axis.
253
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
45
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
45
Routh-Hurwitz Criterion II
201
In the application of the Routh-Hurwitz criterion, two specific scenarios can arise that complicate stability analysis.
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
201
Linear Approximation in Time Domain
70
Nonlinear systems often require sophisticated approaches for accurate modeling and analysis, with state-space representation being particularly effective. This method is especially useful for systems where variables and parameters vary with time or operating conditions, such as in a simple pendulum or a translational mechanical system with nonlinear springs.
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
70
Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
