用于DNN重量修剪的乘数器重量化交替方向方法
Ming Yuan1, Lin Du1, Feng Jiang2
1MIIT Key Laboratory of Dynamics and Control of Complex Systems, Xi'an 710072, China; School of Mathematics and Statistics, Northwestern Polytechnical University, Xi'an 710072, China.
概括
本研究介绍了一种新的动态规范化修剪方法,使用用于深度神经网络 (DNN) 的乘法器 (ADMM) 的交替方向方法. 该技术提高了模型的压缩和准确性,同时降低了计算负载.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 深度神经网络 (DNN) 面临着由于复杂性和尺寸的增加而带来的计算挑战.
- 重量修剪是通过减少模型大小和计算成本来优化DNN的一个关键技术.
- 现有的修剪方法往往缺乏效率或需要广泛的超参数调整.
研究的目的:
- 为DNNs提出一种基于动态规范化的新型修剪方法.
- 将乘数的交替方向方法 (ADMM) 与重量重定机制集成在一起,以改善重量重要性分配.
- 为了减少DNN优化中的计算负担和超参数依赖.
主要方法:
- 开发了一种采用ADMM的动态规范化修剪方法.
- 引入了重权重机制,以动态地赋予网络权重的重要性.
- 在各种DNN架构 (LeNet-5,ResNet-32,ResNet-56,ResNet-50) 和数据集 (MNIST,CIFAR-10,ImageNet) 上评估了该方法.
主要成果:
- 与最先进的修剪方法相比,实现了更高的压缩比率和精度.
- 在LeNet-5 (MNIST) 上展示了显著的压缩 (355.9×),精度得到了改进.
- 在ResNet-50 (ImageNet) 上获得了实质性的压缩 (4.24×),而不会损失精度.
- 展示了降低超参数要求,节省了大量的时间.
结论:
- 提出的基于ADMM的动态调节修剪方法有效优化DNN.
- 重新加权机制增强了重量重要性分配,从而提高了性能.
- 该方法在高效准确的DNN压缩中提供了显著的进步.
相关概念视频
00:57Weighted Mean
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
01:20Vector Operations
Vectors are physical quantities that have both magnitude and direction. The vector operations include addition, subtraction, and scalar multiplication.
A vector multiplied by a scalar value is called scalar multiplication. The result obtained is a new vector with a different magnitude. If the scalar is positive, the direction of the vector remains the same, but if it is negative, the direction of the vector is reversed. For example, the product of the mass and velocity yields the momentum.
A vector multiplied by a scalar value is called scalar multiplication. The result obtained is a new vector with a different magnitude. If the scalar is positive, the direction of the vector remains the same, but if it is negative, the direction of the vector is reversed. For example, the product of the mass and velocity yields the momentum.
01:13Integrator and Differentiator
Op-amp circuits have significant applications in various fields, including automotive engineering. One such application is cruise control systems in cars, where op-amp circuits are integral for maintaining a constant speed. In these systems, op-amps function as both integrators and differentiators.
An integrator within an op-amp circuit produces an output directly proportional to the integral of the input signal. This is achieved by replacing the feedback resistor in a typical inverting...
An integrator within an op-amp circuit produces an output directly proportional to the integral of the input signal. This is achieved by replacing the feedback resistor in a typical inverting...
01:18Reducing Line Loss
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
01:26Linear Approximation in Frequency Domain
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
01:26The Dot Product
Measuring how one directional quantity affects another along a specific path involves comparing their orientation and strength. When two such quantities are represented using direction and amount, a numerical result is computed to show how much one acts along the path of the other. This result comes from a rule combining both inputs' horizontal and vertical parts and adding the results.This calculation gives a single value that grows larger when both inputs point in similar directions and...