重新思考批量规范化和超越批量规范化的有效性
IEEE transactions on pattern analysis and machine intelligence
|September 25, 2023
概括
完整批量规范化 (CBN) 通过诱导关键梯度效应来改善深度学习,在各种激活和数据集中超越标准批量规范化 (BN),以实现更快的融合和更高的准确性.
科学领域:
- 深度学习 (Deep Learning) 是一种深度学习.
- 神经网络优化神经网络优化
背景情况:
- 批量规范化 (BN) 加快了深度神经网络训练,但其有效性机制仍在争论中,最近的重点是Lipschitzness.
- 现有的研究问题是,Lipschitzness是否完全解释了BN的好处,以及 vanila BN是否可以进一步优化.
研究的目的:
- 研究深度神经网络中批量规范化 (BN) 效果的潜在机制.
- 提出和验证一种新的规范化技术,即完整批量规范化 (CBN),以改善训练动态和性能.
主要方法:
- 关于非凸问题上的随机梯度下降 (SGD) 的理论分析,确定了三个关键的趋同增强效应.
- 通过修改BN的结构和规范化位置来开发完整批量规范化 (CBN).
- 经验验证使用CIFAR10,CIFAR100和ILSVRC2012数据集的广泛实验,具有多种激活功能.
主要成果:
- 香草BN与ReLU一起诱导了三种对收的有益效应 (梯度利普希茨常数减小,减少平方随机梯度预期和减少随机梯度变异),但与其他激活并非如此.
- 完全批量规范化 (CBN) 始终产生所有三种融合增强效应,无论使用的激活函数如何.
- CBN显示了更快的训练趋同,导致更小的局部最小值,并显著提高了多个激活函数 (Sigmoid,Tanh,ReLU,SELU,Swish) 的测试准确性,Sigmoid,Tanh和SELU的显著改进.
结论:
- BN的有效性来自于诱导特定的梯度特性,而不仅仅是Lipschitzness.
- 完整批量规范化 (CBN) 提供了理论上有基础的和经验验证的改进,相比香草BN,增强跨多种网络架构和激活的训练稳定性和性能.
- 通过使用传统上效率较低的激活功能,CBN实现了高性能,扩大了深度学习模型的适用性.
相关概念视频
01:52Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
02:07Improving Translational Accuracy
11.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.5K
02:19Buffer Effectiveness
49.1K
Buffer solutions do not have an unlimited capacity to keep the pH relatively constant . Instead, the ability of a buffer solution to resist changes in pH relies on the presence of appreciable amounts of its conjugate weak acid-base pair. When enough strong acid or base is added to substantially lower the concentration of either member of the buffer pair, the buffering action within the solution is compromised.
The buffer capacity is the amount of acid or base that can be added to a given volume...
The buffer capacity is the amount of acid or base that can be added to a given volume...
49.1K
01:13Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
80
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
80