在神经网络上,通过信息瓶类似的方法,拟合,压缩和泛化行为
Zhaoyan Lyu1, Gholamali Aminian2, Miguel R D Rodrigues1
1Department of Electronic and Electrical Engineering, University College London, Gower St., London WC1E 6BT, UK.
Entropy (Basel, Switzerland)
|July 29, 2023
概括
本研究引入了一种新方法,通过用更简单的措施取代复杂的相互信息来理解神经网络学习动态. 这种方法可靠地捕捉了适配和压缩阶段,这对于概括至关重要.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 信息理论 信息理论
背景情况:
- 神经网络的学习动态,包括配合,压缩和概括,仍然不太了解.
- 对于神经网络来说,在高维空间中估计相互信息是具有挑战性的.
研究的目的:
- 提出一种用于捕捉神经网络学习动态的新方法.
- 将难以估计的相互信息措施替换为更容易处理的相互信息措施.
主要方法:
- 使用了信息瓶类型的技术.
- 取代了相互信息,用于输入重建的最小平均平方误差和分类的交叉.
- 通过各种网络模型,学习算法和数据集进行实证研究.
主要成果:
- 拟议的方法可靠地捕捉神经网络在训练和测试期间的动态,优于传统的信息瓶方法.
- 在不同的激活函数中观察到一致的安装和压缩阶段.
- 更好的概括与更明显的拟合和压缩相相关.
结论:
- 这种新的方法提供了一种更可靠的方法来分析神经网络的学习动态.
- 拟合和压缩是神经网络学习的基本阶段,受架构,训练和数据的影响.
- 了解这些阶段是改善模型概括的关键.
相关概念视频
Generalization, Discrimination, and Extinction
620
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
620
Survival Tree
111
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
111
Neuroplasticity
569
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
569
Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
81
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
81
Parallel Processing
182
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
182


