WideTopo:通过培训动态的保存和广泛的拓学的探索来改善预测神经网络的修剪
Changjian Deng1, Jian Cheng1, Yanzhou Su1
1School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, PR China.
概括
WideTopo通过保持训练动态和维护层宽度来修剪神经网络,提高子网络的稳定性和性能. 这种方法可以提高计算效率而不会牺牲准确性,特别是在转移学习中.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 神经网络修剪对于计算资源效率至关重要.
- 现有的方法,如基于突出度和基于图形理论的修剪有局限性.
- 突出度得分可以导致狭窄的子网络;图表方法与预先训练的模型作斗争.
研究的目的:
- 提出WideTopo,一种新的修剪方法,可以识别结构稳定的子网络.
- 通过保持培训动态和探索广泛的网络拓来提高培训后的绩效.
- 解决现有的修剪技术的局限性,特别是在转移学习中.
主要方法:
- WideTopo集成了神经触点内核 (NTK) 理论和隐式目标对齐 (ITA) 来捕捉子网络训练动态.
- 采用密度意识突出度得分衰减策略来保留有效的节点.
- 使用重复面具恢复策略来维持子网络中的层宽度.
主要成果:
- 在CNN和ViT模型上进行了广泛的验证,用于图像分类和语义细分.
- 通过随机和预训练的初始化,在各种架构和模型密度中证明了有效性.
- 与现有的基线相比,WideTopo实现了竞争性的培训后表现.
结论:
- WideTopo有效地识别了结构稳定的子网络,并提高了性能.
- 该方法适用于各种架构和初始化设置,性能优于当前的基线.
- 保持训练动态和保持网络宽度是有效神经网络修剪的关键.
相关概念视频
01:01Neuroplasticity
1.6K
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
1.6K
01:19Survival Tree
388
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
388