Unified knowledge distillation: Integrating offline, online, and self-distillation paradigms for image classification

Knowledge distillation (KD) can be broadly categorized into three paradigms: offline KD, online KD, and self-KD. However, these individual paradigms typically suffer from respective limitations in terms of adaptability, stability, and expressiveness. Meanwhile, their effective integration remains challenging due to heterogeneous supervision and the additional complexity introduced by multi-paradigm collaboration. To address these issues, we propose UKD, a Unified Knowledge Distillation framework that explicitly coordinates the three heterogeneous paradigms within a single optimization process for efficient knowledge transfer across diverse architectures and model capacities. Specifically, UKD introduces a lightweight proxy teacher to bridge the three paradigms and reduce capacity gap, which inherits the smoothed core knowledge from the pre-trained primary teacher, drives student-oriented knowledge adaptation and guides the internal consistency of student model. Moreover, UKD designs a dual-teacher collaboration strategy that dynamically adjusts teaching intensity of the powerful primary teacher and capacity-reduced proxy teacher based on the learning state of the student to improve the capacity adaptation of the model. To ensure efficiency, UKD further adopts backbone-sharing mechanisms with lightweight auxiliary modules to reduce training overhead. Extensive experiments on CIFAR-100 and ImageNet-1K across CNN, VIT, and MLP architectures demonstrate the effectiveness and efficiency of UKD, achieving gains of up to 11.38% and 2.68% with modest overhead, respectively. Code is available at https://anonymous.4open.science/r/UKD-ECD6.