细粒度对于智能神经网络修剪至关重要.
Alex Heyman1, Joel Zylberberg2
1Department of Electrical Engineering and Computer Science, York University, Toronto, ON, CA M3J 1P3 aheyman@yorku.ca.
Neural computation
|October 9, 2024
概括
智能神经网络修剪对随机修剪的精度优势随着颗粒度变得更粗而减少. 细粒度重量修剪提供了精度和计算成本降低的最佳平衡.
科学领域:
- 人工智能的人工智能
- 计算机科学 计算机科学
- 机器学习 机器学习
背景情况:
- 神经网络修剪降低了计算成本,但可能会影响准确性.
- 修剪方法的颗粒度各不相同,从单个重量 (细) 到结构部件 (粗).
- 细粒度修剪删除了更多的参数,而粗粒度修剪更好地利用了硬件优化.
研究的目的:
- 综合调查修剪颗粒度与智能修剪与随机修剪相比的有效性之间的关系.
- 在各种细粒度和架构中评估不同的智能修剪标准.
- 确定最佳的修剪策略,以平衡精度和计算效率.
主要方法:
- 将智能代修剪与初始化时的随机修剪进行比较.
- 在两个架构上评估了多个修剪颗粒度 (重量,内核,通道).
- 测试了三个图像分类任务的性能,测量精度和训练代.
主要成果:
- 智能比随机修剪的精度优势随着粗细粒度的增加而显著下降.
- 在高的修剪速度下,细粒度重量修剪保持了接近未修剪的精度,而粗修剪比随机修剪产生了最小的收益.
- 对于CIFAR-10上的ResNet-20,智能重量修剪实现了90.0%的准确性,内核修剪86.5%,道修剪85.5%,与随机修剪的85.0%相比.
结论:
- 精细的修剪,当与高效的实现相结合时,更有希望在低计算成本下实现高精度.
- 在准确性保护方面,粗粒度修剪在随机修剪上提供了有限的优势.
- 选择颗粒度对智能神经网络修剪策略的有效性产生了重大影响.
相关概念视频
01:01Neuroplasticity
303
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
303
01:19Survival Tree
61
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
61