Heterogeneous Mutual Knowledge Distillation for Wearable Human Activity Recognition.

Area of Science:

• Computer Science
• Machine Learning
• Wearable Technology

Background:

• Deep learning models for human activity recognition (HAR) face challenges in efficient knowledge transfer to lightweight mobile devices.
• Existing knowledge distillation (KD) methods primarily focus on homogeneous model architectures, limiting their application in heterogeneous wearable HAR scenarios.

Purpose of the Study:

• To propose a novel heterogeneous mutual knowledge distillation (HMKD) framework specifically designed for wearable human activity recognition.
• To address the limitations of existing KD techniques in heterogeneous model setups for efficient model compression.

Main Methods:

• Developed a heterogeneous mutual KD (HMKD) framework enabling mutual learning between intermediate and output layers of teacher and student models.
• Implemented a weighted ensemble feature approach to merge intermediate features, facilitating knowledge exchange despite significant structural differences between heterogeneous models.

Main Results:

• HMKD demonstrated superior performance compared to ten state-of-the-art KD algorithms across HAPT, WISDM, and UCI_HAR datasets, particularly in classification accuracy.
• A notable improvement of 9.19% in the $F_{1}$ score for the MLP student model was achieved when using ResNetLSTMaN as the teacher on the HAPT dataset.

Conclusions:

• The proposed HMKD framework effectively bridges the gap in heterogeneous knowledge transfer for wearable HAR.
• HMKD offers a promising solution for compressing complex HAR models into lightweight versions for mobile deployment without sacrificing performance.