基于邻居关系的知识蒸用于图像分类
Jianping Gou1, Xiaomeng Xin2, Baosheng Yu3
1College of Computer and Information Science, College of Software, Southwest University, Chongqing, 400715, China.
概括
本研究介绍了基于邻近关系的知识蒸 (NRKD),这是一个新的模型压缩方法. 通过考虑样本关系,NRKD改善了知识转移,优于现有的蒸技术.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 知识蒸是一种关键的模型压缩技术,将知识从大型教师模型转移到较小的学生模型.
- 现有的关系蒸方法往往忽略了单个数据样本之间的细微关系.
- 目前的方法通常依赖于随机抽样选择,可能缺少关键的本地结构信息.
研究的目的:
- 引入基于邻近关系的知识蒸 (NRKD) 作为模型压缩中知识转移的先进方法.
- 利用数据中的本地结构和样本间的关系来实现更有效的蒸.
- 通过结合新的关系知识来提高学生模型的表现.
主要方法:
- NRKD根据小批量相似性矩阵确定了选定样品的K-最近邻居.
- 它构建了邻居关系知识,捕获了当地的数据结构.
- 这种关系知识是通过中间特征图和输出逻辑传递的.
主要成果:
- 在CIFAR10,CIFAR100,Tiny ImageNet和ImageNet数据集上进行了广泛的实验.
- 与最先进的知识蒸方法相比,NRKD表现出了竞争力.
- 拟议的方法有效地转移关系知识,以提高学生模型的准确性.
结论:
- 基于邻近关系的知识蒸 (NRKD) 为模型压缩提供了一种新且有效的方法.
- 考虑本地样本结构显著增强了蒸中的知识转移.
- 在高效的深度学习模型领域,NRKD提供了宝贵的进步.
相关概念视频
Aggregates Classification
292
Aggregate classification is generally based on its size, petrographic characteristics, weight, and source. Size classification ranges from coarse to fine aggregates, defined by the size of the particles. Coarse aggregates are particles that do not pass through ASTM sieve No. 4, and aggregates that pass through the sieve are fine aggregates.
Petrographic classification groups aggregates based on common mineralogical characteristics. Some of the common mineral groups found in aggregates are...
Petrographic classification groups aggregates based on common mineralogical characteristics. Some of the common mineral groups found in aggregates are...
292
Classification of Systems-II
129
Continuous-time systems have continuous input and output signals, with time measured continuously. These systems are generally defined by differential or algebraic equations. For instance, in an RC circuit, the relationship between input and output voltage is expressed through a differential equation derived from Ohm's law and the capacitor relation,
129
Classification of Systems-I
161
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
161
Generalization, Discrimination, and Extinction
355
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
355
Force Classification
1.1K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.1K
Associative Learning
255
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
255


