高维回归的信息瓶理论:相关性,效率和最佳性
Vudtiwat Ngampruetikorn1, David J Schwab1
1Initiative for the Theoretical Sciences, The Graduate Center, CUNY.
Advances in neural information processing systems
|June 19, 2023
概括
研究人员使用剩余信息量化了机器学习中的过拟合. 最优的算法将这种噪音降到最低,平衡相关信息以获得更好的预测和理解信息效率.
科学领域:
- 机器学习 机器学习
- 信息理论 信息理论
- 统计建模 统计建模
背景情况:
- 过度装配是机器学习的一个重大挑战,在机器学习中,模型学习训练数据噪声.
- 大型神经网络经常实现零训练损失,这与过度配合产生了令人费解的矛盾.
- 需要新的方法来理解和量化复杂模型中的过拟合.
研究的目的:
- 用剩余信息的概念来量化过.
- 调查剩余信息 (噪音) 和相关信息 (预测信号) 之间的权衡.
- 分析学习算法的信息效率,特别是随机回归,与最佳算法相比.
主要方法:
- 通过剩余信息 (比特编码训练数据噪声) 定义和量化过拟合.
- 制定了一个优化问题,以确定信息效率高的学习算法.
- 解决了线性回归的优化,并将结果与随机回归进行比较.
- 应用随机矩阵理论来分析高维线性地图学习.
主要成果:
- 在机器学习模型中证明了剩余和相关信息之间的基本权衡.
- 描述了随机回归相对于理论上最佳算法的信息效率.
- 揭示了学习高维线性地图的信息复杂性.
- 确定了双重和多重下降现象的信息理论类型.
结论:
- 剩余信息提供了一种有原则的方法来量化过.
- 信息效率为设计更好的学习算法提供了一个框架.
- 了解信息复杂性对于高维学习至关重要,并解释了像双降落这样的现象.
相关概念视频
Column Efficiency: Rate Theory
387
The rate theory of chromatography provides quantitative insight into the shapes and widths of elution bands. These bands are based on the random-walk mechanism governing molecular migration within a column. The Gaussian profile of chromatographic bands arises from the cumulative effect of random molecular motions as they progress through the column.
During elution, a solute molecule experiences numerous transitions between stationary and mobile phases, exhibiting irregular residence times in...
During elution, a solute molecule experiences numerous transitions between stationary and mobile phases, exhibiting irregular residence times in...
387
Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
Multiple Regression
3.0K
Multiple regression assesses a linear relationship between one response or dependent variable and two or more independent variables. It has many practical applications.
Farmers can use multiple regression to determine the crop yield based on more than one factor, such as water availability, fertilizer, soil properties, etc. Here, the crop yield is the response or dependent variable as it depends on the other independent variables. The analysis requires the construction of a scatter plot...
Farmers can use multiple regression to determine the crop yield based on more than one factor, such as water availability, fertilizer, soil properties, etc. Here, the crop yield is the response or dependent variable as it depends on the other independent variables. The analysis requires the construction of a scatter plot...
3.0K
The Availability Heuristic
6.0K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
6.0K
Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
84
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
84


