线性递归特征机器可以证明可以恢复低等级矩阵
Adityanarayanan Radhakrishnan1,2, Mikhail Belkin3, Dmitriy Drusvyatskiy4
1Applied Math, Harvard University, MA 02138.
概括
递归特征机器 (RFM) 为理解神经网络特征学习提供了一种新方法. 这种方法明确执行维度减小,在稀疏的恢复任务中表现优于深度线性网络.
科学领域:
- 机器学习 机器学习
- 统计推理 统计推理
- 计算理论 计算理论
背景情况:
- 神经网络在预测方面表现出色,似乎挑战了维度的诅咒.
- 特征学习,一种维度缩小形式,是这种成功的假设原因.
- 平均梯度外积 (AGOP) 是一个与特征学习相关的统计估计器.
研究的目的:
- 在维度缩小方面为递归特征机器 (RFM) 提供理论保证.
- 将神经网络中的特征学习与经典的稀疏恢复算法连接起来.
- 为实际应用开发可扩展和高效的RFM实现.
主要方法:
- 对稀疏线性回归和低级矩阵恢复中的过度参数化的问题分析递归特征机器 (RFM).
- 证明线性RFM (lin-RFM) 相当于代重量最小方程 (IRLS) 的一种变体.
- 实施lin-RFM来处理大量缺失条目的大型矩阵.
主要成果:
- 建立了RFM的维度减少能力的理论保证.
- 已经证明Lin-RFMs是IRLS算法的一个变体.
- 开发的lin-RFM实现速度比标准IRLS快,性能优于深线性网络.
结论:
- RFM提供了明确的特征学习,为神经网络行为提供了洞察力.
- 这项研究弥合了神经网络特征学习和经典稀疏恢复之间的差距.
- 高效的lin-RFM实现使其能够在大规模稀疏回收和矩阵完成方面得到实际应用.
相关概念视频
Routh-Hurwitz Criterion I
Consider an electrical power grid, where stability is essential to prevent blackouts. The Routh-Hurwitz criterion is a valuable tool for assessing system stability under varying load conditions or faults. By analyzing the closed-loop transfer function, the Routh-Hurwitz criterion helps determine whether the system remains stable.
To apply the Routh-Hurwitz criterion, a Routh table is constructed. The table's rows are labeled with powers of the complex frequency variable s, starting from the...
To apply the Routh-Hurwitz criterion, a Routh table is constructed. The table's rows are labeled with powers of the complex frequency variable s, starting from the...
Routh-Hurwitz Criterion II
In the application of the Routh-Hurwitz criterion, two specific scenarios can arise that complicate stability analysis.
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first column of the Routh...
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first column of the Routh...
Sequence Networks of Rotating Machines
A Y-connected synchronous generator, grounded through a neutral impedance, is designed to produce balanced internal phase voltages with only positive-sequence components. The generator's sequence networks include a source voltage that is exclusively in the positive-sequence network. The sequence components of line-to-ground voltages at the generator terminals illustrate this configuration.
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...
Linearization and Approximation
Linearization is a mathematical technique used to approximate complex, nonlinear functions with simpler linear models in the vicinity of a chosen reference point. The method is based on the idea that, although a function may be difficult to evaluate exactly, its behavior near a specific input value can often be closely approximated by the tangent line at that point. This approach is particularly useful when small deviations from a known value are involved.Consider the square root function, for...
Application of Linearization and Approximation
A drone flying through complex terrain often relies on more than one sensing method to estimate small changes in altitude. Along with direct measurements, air pressure provides a useful indirect indicator of vertical movement. Atmospheric pressure decreases as altitude increases, and this relationship is commonly described using an exponential model. Although accurate, converting pressure measurements into altitude values requires calculations that are too complex to perform repeatedly during...
Linear Approximations
For a differentiable function of two variables, linear approximation estimates values near a known point by replacing the curved surface with its tangent plane. Consider the function\begin{equation*}f(x,y)=x^2+3y^2\end{equation*}near the point (2, 1). The exact value at this point is f(2, 1) = 22 + 3(1)2 = 4 + 3 = 7.The linear approximation of f(x, y)) near (a, b) is\begin{equation*}L(x,y)=f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)\end{equation*}First, compute the partial derivatives: fx(x, y) = 2x and...

