双向自我规范化的神经网络
Yao Lu1, Stephen Gould2, Thalaiyasingam Ajanthan3
1Australian National University, Australia; Peking University, China.
概括
神经网络训练通过解决消失和爆炸梯度而得到改善. 使用高斯-波因卡雷正常化和直角权重的新方法确保在深度网络中稳定的信号传播.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算神经科学是一种神经科学.
背景情况:
- 消失和爆炸的梯度是训练深度神经网络的持续挑战.
- 现有的实际解决方案缺乏严格的理论基础.
- 高维概率理论为分析提供了一个潜在的框架.
研究的目的:
- 理论上解决神经网络中消失和爆炸梯度的问题.
- 提出一种稳定信号传播的新方法.
- 用经验证据验证拟议的方法.
主要方法:
- 使用高维概率理论来分析梯度流.
- 引入高斯-波恩卡雷正常化函数作为一个新类的激活函数.
- 使用直角重量矩阵来限制信号传播.
- 在合成和现实数据集上进行实验.
主要成果:
- 证明在温和条件下,足够的网络宽度可以缓解消失/爆炸梯度的高概率.
- 展示了高斯-波因卡雷正常化和直角重量矩阵在稳定前向和后向信号传播方面的有效性.
- 经验验证证证实了关于非常深层神经网络的理论发现.
结论:
- 提出的理论框架和实践方法有效解决了消失/爆炸梯度的问题.
- 这种方法提高了非常深层的神经网络的可训练性.
- 这项工作为深度学习的长期挑战提供了严格的,可证明的解决方案.
相关概念视频
01:25Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K
01:37Neural Regulation
39.5K
Digestion begins with a cephalic phase that prepares the digestive system to receive food. When our brain processes visual or olfactory information about food, it triggers impulses in the cranial nerves innervating the salivary glands and stomach to prepare for food.
39.5K