SMKD:选择性的相互知识蒸
Ziyun Li1, Xinshao Wang2, Neil M Robertson3
1Hasso Plattner Institute, University of Potsdam, Potsdam, Germany.
概括
选择性相互知识蒸 (SMKD) 通过过不准确的信息来提高模型可靠性. 这种方法通过选择可靠的知识进行蒸来改善知识传输,特别是在处理杂数据时.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 计算机科学 计算机科学
背景情况:
- 相互知识蒸 (MKD) 促进了模型之间的协作知识传输.
- 不可靠的知识,特别是来自杂的标签,可以导致模型记住不正确的信息.
- 选择性知识蒸比一般模型可靠性更少被探索.
研究的目的:
- 引入选择性相互知识蒸 (SMKD) 的新框架.
- 为了应对MKD中不可靠的知识转移的挑战.
- 为MKD提供统一的框架,包括各种知识选择策略.
主要方法:
- 开发了SMKD的通用知识选择公式.
- 在SMKD框架内实施了静态和渐进的选择值.
- 设计的SMKD包括使用没有或全部知识的特殊情况,统一MKD方法.
主要成果:
- 通过广泛的实验证明了拟议的SMKD框架的有效性.
- 验证了选择性知识选在相互蒸中的重要性.
- 展示了SMKD在具有挑战性的条件下提高模型性能的能力.
结论:
- SMKD提供了一个强大的解决方案,以提高知识蒸的可靠性.
- 拟议的框架为知识选择策略提供了灵活性.
- SMKD代表了选择性知识蒸领域的重大进步.
相关概念视频
Self-Schemas
31.0K
In general, a schema is a mental construct consisting of a cluster or collection of related concepts (Bartlett, 1932). There are many different types of schemata, and they all have one thing in common: schemata are a method of organizing information that allows the brain to work more efficiently. When a schema is activated, the brain makes immediate assumptions about the person or object being observed.
31.0K
Nonconscious Mimicry
4.5K
Nonconscious mimicry occurs when individuals alter their mannerisms to match the behaviors and expressions of those nearby, without intention.
4.5K
Cognitive Learning
229
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
229
Hindsight Biases
3.4K
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
3.4K
Relationship Formation
39.9K
What do you think is the single most influential factor in determining with whom you become friends and whom you form romantic relationships? You might be surprised to learn that the answer is simple: the people with whom you have the most contact. This most important factor is proximity. You are more likely to be friends with people you have regular contact with. For example, there are decades of research that shows that you are more likely to become friends with people who live in your dorm,...
39.9K
The Representativeness Heuristic
15.8K
The representative heuristic describes a biased way of thinking, in which you unintentionally stereotype someone or something. For example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation, because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.
15.8K


