转移KD: 在分配转移下对比知识蒸
Songming Zhang1, Yuxiao Luo2, Ziyu Lyu3
1School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China; Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology, Shenzhen, China.
概括
本研究介绍了ShiftKD,这是一个基准框架,用于在分配转移下评估知识蒸 (KD) 方法. 它揭示了当前KD技术的局限性,并指导开发更强大的模型,用于现实世界的应用.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 模型压缩压缩模型
背景情况:
- 知识蒸 (KD) 成功地将知识从大模型转移到小模型.
- 在分配转移下KD方法的可靠性未得到充分探索.
- 分布转移,在训练和测试数据分布不同的地方,可以降低KD性能.
研究的目的:
- 提出一个统一的框架,ShiftKD,用于对分布转移进行KD方法的基准测试.
- 在多样性和相关性转移下系统评估KD性能.
- 确定影响KD学生模式培训的关键因素.
主要方法:
- 开发了ShiftKD,一个全面的评估基准.
- 包括超过30个KD方法跨算法,数据驱动和优化方法.
- 利用五个基准数据集来评估分配转移下的表现.
主要成果:
- 进行了广泛的实验,揭示了最先进的KD方法的优点和局限性.
- 分析了数据增强,修剪,优化器和评估指标对学生模型培训的影响.
- 确定了强大的KD性能的关键因素.
结论:
- ShiftKD提供了一个有效的基准来评估KD可靠性在现实世界的场景.
- 这些发现将推动开发更强大的KD方法,适应分布变化.
- 这项工作促进了模型压缩和部署的进步.
相关概念视频
Uniform Distribution
5.2K
The uniform distribution is a continuous probability distribution of events with an equal probability of occurrence. This distribution is rectangular.
Two essential properties of this distribution are
Two essential properties of this distribution are
5.2K
Probability Distributions
7.9K
The probability of a random variable x is the likelihood of its occurrence. A probability distribution represents the probabilities of a random variable using a formula, graph, or table. There are two types of probability distribution– discrete probability distribution and continuous probability distribution.
A discrete probability distribution is a probability distribution of discrete random variables. It can be categorized into binomial probability distribution and Poisson...
A discrete probability distribution is a probability distribution of discrete random variables. It can be categorized into binomial probability distribution and Poisson...
7.9K
Distribution and Dispersion
22.4K
To understand intra-specific interactions in populations, scientists measure the spatial arrangement of species individuals. This geographic arrangement is known as the species distribution or dispersion. Highly territorial species exhibit a uniform distribution pattern, in which individuals are spaced at relatively equal distances from one another. Species that are highly tied to particular resources, such as food or shelter, tend to concentrate around those resources, and thus exhibit a...
22.4K
Normal Distribution
12.5K
The normal, a continuous distribution, is the most important of all the distributions. Its graph is a bell-shaped symmetrical curve, which is observed in almost all disciplines. Some of these include psychology, business, economics, the sciences, nursing, and, of course, mathematics. Some instructors may use the normal distribution to help determine students’ grades. Most IQ scores are normally distributed. Often real-estate prices fit a normal distribution. The normal distribution is...
12.5K
Extraction: Partition and Distribution Coefficients
2.9K
The distribution law or Nernst's distribution law is the law that governs the distribution of a solute between two immiscible solvents. This law, also known as the partition law, states that if a solute is added to the mixture of two immiscible solvents at a constant temperature, the solute is distributed between the two solvents in such a way that the ratio of solute concentrations in the solvents remains constant at equilibrium.
For extracting a solute from an aqueous phase into an...
For extracting a solute from an aqueous phase into an...
2.9K
Data: Types and Distribution
844
In biostatistics, data are the observations collected for analysis. There are two main types: parametric and non-parametric. Parametric data, which include continuous (e.g., weight) and discrete numerical data (e.g., number of tablets), assume a particular distribution pattern, often the normal distribution. Non-parametric data do not adhere to a specific distribution and typically comprise nominal (e.g., gender) and ordinal categorical data (e.g., pain scale ratings).
Distributions in...
Distributions in...
844


