Multiscale knowledge distillation with attention based fusion for robust human activity recognition

Area of Science:

• Machine Learning
• Computer Vision
• Artificial Intelligence

Background:

• Knowledge distillation is crucial for training multi-modal models with asynchronous data.
• Existing methods struggle with comprehensive cross-modal and cross-model knowledge transfer.

Purpose of the Study:

• To propose a novel multiscale knowledge distillation framework.
• To enhance knowledge transfer efficiency and model robustness in multi-modal learning.

Main Methods:

• Introduced a multiscale semantic graph mapping (SGM) loss function for detailed knowledge transfer.
• Developed a fusion and tuning (FT) module to leverage intra- and inter-modal correlations.
• Utilized transformer-based backbones for advanced feature learning.

Main Results:

• Achieved performance improvements of 2.31% on the MMAct dataset and 0.29% on the UTD-MHAD dataset for multimodal human activity recognition.
• Ablation studies confirmed the effectiveness and necessity of each proposed component.

Conclusions:

• The proposed multiscale knowledge distillation framework effectively addresses limitations in traditional methods.
• The framework demonstrates superior performance in multimodal human activity recognition tasks.