神经网络中疏忽的分析特征:从线性模型的洞察力
Jialin Mao1, Itay Griniasty2, Yan Sun1
1University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA.
Physical review. E
|February 20, 2026
概括
深度神经网络训练遵循一个低维的多重组. 这项研究分析地描述了线性网络中的这种"超带"现象,确定了关键控制因素,如数据相关性和初始权重.
科学领域:
- 机器学习 机器学习
- 动态系统理论 动态系统理论
- 计算神经科学是一种神经科学.
背景情况:
- 深度神经网络在不同的环境中表现出一致的训练轨迹.
- 这些轨迹似乎局限于概率分布空间中的一个低维多重体,称为"超大带".
- 深度网络中这种现象的根本原因仍然是一个活跃的研究领域.
研究的目的:
- 在深度神经网络训练中观察到的低维多元组 (超大带) 的分析特征.
- 在更简单的模型中调查控制几何和这种多元体出现的因素.
- 将这些分析见解扩展到更广泛的机器学习模型类别.
主要方法:
- 应用动态系统理论来分析训练动态.
- 线性网络中的训练轨迹的分析特征.
- 从关键控制参数计算和限制贡献.
主要成果:
- 线性网络中的超带多元体的几何是由输入相关性矩阵的固有值衰变,初始重量对输出尺度和梯度下降步骤决定的.
- 分析计算的阶段边界的超级丝带的形成.
- 将分析扩展到以随机梯度下降训练的内核机器和线性模型.
结论:
- 深度学习中的超带现象起源于线性模型的动态.
- 了解这些低维的多元体,可以了解神经网络的概括能力.
- 分析框架提供了一种预测和潜在控制网络训练行为的方法.
相关概念视频
Classification of Systems-I
742
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
742
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
438
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
438
Nonlinear Pharmacokinetics: Causes of Nonlinearity
935
Nonlinearity in drug pharmacokinetics is caused by various factors influencing how a drug is absorbed, distributed, metabolized, and excreted. Understanding these nonlinear processes is crucial for predicting drug behavior in the body and optimizing drug dosing regimens.
Nonlinear drug absorption can occur when the process is rate-limited by solubility, carrier-mediated transport systems, or saturation of the presystemic gut wall or hepatic metabolism. For instance, high doses of riboflavin...
Nonlinear drug absorption can occur when the process is rate-limited by solubility, carrier-mediated transport systems, or saturation of the presystemic gut wall or hepatic metabolism. For instance, high doses of riboflavin...
935
Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
427
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
427
Linearization and Approximation
233
Linearization is a mathematical technique used to approximate complex, nonlinear functions with simpler linear models in the vicinity of a chosen reference point. The method is based on the idea that, although a function may be difficult to evaluate exactly, its behavior near a specific input value can often be closely approximated by the tangent line at that point. This approach is particularly useful when small deviations from a known value are involved.Consider the square root function, for...
233
Application of Linearization and Approximation
193
A drone flying through complex terrain often relies on more than one sensing method to estimate small changes in altitude. Along with direct measurements, air pressure provides a useful indirect indicator of vertical movement. Atmospheric pressure decreases as altitude increases, and this relationship is commonly described using an exponential model. Although accurate, converting pressure measurements into altitude values requires calculations that are too complex to perform repeatedly during...
193

