在自我蒸中重组教师和学生
概括
重组自蒸 (RSD) 通过重组教师和学生网络来增强模型压缩. 这种新的自蒸框架通过增加建筑的多样性并实现有效的知识传输来提高性能.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 知识蒸是模型压缩的关键,将知识从大模型转移到小模型.
- 自蒸方法使用模型自己的知识,但受到类似的教师-学生架构的限制.
- 现有的自蒸技术与建筑的多样性作斗争,阻碍了最佳的性能.
研究的目的:
- 引入一个新的自蒸框架,重组自蒸 (RSD),以增强模型压缩.
- 通过增加建筑多样性来克服类似网络架构在自蒸中的局限性.
- 通过建筑结构重组,提高自蒸知识转移的有效性.
主要方法:
- 拟议的重组自蒸 (RSD) 框架重组了教师和学生网络.
- 教师模型被扩展为多分支拓,通过抛弃教师分支,创建多样化的学生子网络.
- 一个特征混合块被随机插入,以链接教师和学生模型,以促进混合特征空间的蒸. 教师分支将重新转换为原始结构,以避免额外的推断成本.
主要成果:
- 综合实验证明了RSD框架在各种架构中的有效性.
- 拟议的方法显示了对标准数据集 (如CIFAR-10/100和ImageNet.Net) 的显著改进.
- 与现有的自蒸方法相比,RSD成功地提高了模型压缩和知识传输.
结论:
- 重组自蒸 (RSD) 为模型压缩和自蒸提供了一种强大的新方法.
- 框架增加建筑多样性的能力对于提高自蒸性能至关重要.
- 在不增加推断成本的情况下,RSD为实现高性能模型压缩提供了有效的解决方案.
相关概念视频
Cognitive Dissonance
32.5K
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
32.5K
Self-Evaluation: Self-Enhancement and Self-Verification
5.2K
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
5.2K
Deindividuation
26.3K
Deindividuation is a form of social influence on an individual’s behavior such that the individual engages in unusual or non-normal behavior while in a group setting. Why? Because in these group settings, the individual no longer sees themselves as an individual anymore, disinhibiting their behavior and personal restraint.
26.3K
Self-Presentation: Self-Monitoring and Self-Handicapping
38.9K
People can go to great lengths to protect their self-image and present themselves in ways that they want others to see them. Sociologist Erving Goffman presented the idea that a person is like an actor on a stage. Calling his theory dramaturgy, Goffman believed that we use “impression management” to present ourselves to others as we hope to be perceived. Each situation is a new scene, and individuals perform different roles depending on who is present (Goffman, 1959). Think about...
38.9K
Self-Discrepancy Theory
18.3K
One influential perspective on what motivates people's behavior is detailed in Tory Higgin's self-discrepancy theory (Higgins, 1987). He proposed that people hold disagreeing internal representations of themselves that lead to different emotional states.
18.3K
The Sense of Self: Reflected Self-Appraisal and Social Comparison
49.7K
According to Charles Cooley, we base our image on what we think other people see (Cooley 1902). We imagine how we must appear to others, then react to this speculation. We don certain clothes, prepare our hair in a particular manner, wear makeup, use cologne, and the like—all with the notion that our presentation of ourselves is going to affect how others perceive us. We expect a certain reaction, and, if lucky, we get the one we desire and feel good about it. But more than that, Cooley...
49.7K


