不要忘记我:用知识融合和蒸来打击当地过度装备

Uri Stern, Eli Corn, Daphna Weinshall

IEEE transactions on pattern analysis and machine intelligence

|December 24, 2025

概括

研究人员在深度学习模型中发现了"局部过拟合",在特定数据区域的性能下降. 一种新的方法恢复了这种被遗忘的知识,提高了模型性能而不会增加复杂性.

科学领域:

人工智能的人工智能
机器学习机器学习
深度学习 (Deep Learning) 是一种深度学习.

背景情况:

深度神经网络 (DNN) 比理论预测表明的过度匹配的情况要少.
传统的过度装配,即随着产能增加而导致的全球性能下降,在实践中很少观察到.
该研究调查了在特定数据子区域发生的过,称为局部过.

研究的目的:

引入一个新的分数来衡量DNN在验证数据上的遗忘率.
定义和量化特定输入空间区域的局部过度装配作为性能退化.
探索局部过和双降现象之间的联系.

主要方法:

开发了一个新的分数来量化验证数据的遗忘率.
提出了一种两阶段的方法:将检查点聚合到一个集体中,然后进行知识蒸.
利用单个模型的训练历史来恢复被遗忘的知识.

主要成果:

证明局部过可以独立于常规过发生.
显示了局部过和双下降现象之间的强烈相关性.
建议的知识融合,然后是知识蒸方法,在不增加推断成本的情况下提高了性能,超过了基线,特别是标签噪音.

结论:

局部过度装配是一个与全球过度装配截然不同的现象,与模型能力和培训动态有关.
一种新的两阶段方法有效地恢复和保留模型培训历史中的遗忘知识.
这种方法提供了更好的性能和更少的复杂性,为深度学习模型优化提供了一个双赢的场景.

相关概念视频

01:19

Survival Tree

369

Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...

369

01:24

Generalization, Discrimination, and Extinction

1.3K

Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...

1.3K

01:22

Force Classification

2.2K

Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...

2.2K

01:12

Observational Learning

782

Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...

782

02:07

Improving Translational Accuracy

14.0K

Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...