Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Traditional knowledge distillation methods are teacher-centered, potentially transferring suboptimal knowledge due to network structure differences.
• Existing methods face challenges like deficiency knowledge, hard sample learning, and knowledge forgetting in student networks.

Purpose of the Study:

• To propose a Student-Centered Distillation (SCD) method inspired by human educational principles.
• To enable teacher networks to adapt knowledge transfer based on student network requirements.
• To address key learning challenges faced by student networks during distillation.

Main Methods:

• Implemented SCD by having the teacher network identify and learn student-desired knowledge on a validation set, then transfer it via the training set.
• Adapted Proportional-Integral-Derivative (PID) algorithms to identify the student network's current knowledge needs.
• Introduced a curriculum learning-based fuzzy strategy integrated with PID control for focused learning on challenging samples.

Main Results:

• Student-Centered Distillation (SCD) demonstrated superior performance compared to existing teacher-centered methods.
• Experimental results across multiple tasks validated the effectiveness of the proposed SCD approach.
• The adapted PID algorithms and fuzzy strategy successfully addressed knowledge deficiency and hard sample learning.

Conclusions:

• Student-Centered Distillation (SCD) offers a more effective knowledge transfer paradigm than traditional teacher-centered approaches.
• The integration of human educational wisdom and automation algorithms enhances student network learning.
• SCD provides a promising direction for future research in knowledge distillation.