Improved Knowledge Distillation Based on Global Latent Workspace With Multimodal Knowledge Fusion for Understanding...

Area of Science:

• Biomedical Engineering
• Machine Learning
• Cognitive Neuroscience

Background:

• Wearable sensors are crucial for health monitoring, but deep learning models struggle with noisy, inconsistent data.
• Topological Data Analysis (TDA) offers robust feature extraction via persistence images (PIs), yet is computationally expensive.
• Knowledge Distillation (KD) enables smaller models but faces challenges with multimodal data and conflicting teacher knowledge.

Purpose of the Study:

• To propose a novel KD framework, multimodal global latent workspace-based KD (mGLW-KD), inspired by cognitive neuroscience's Global Workspace Theory (GWT).
• To address limitations in KD for wearable sensor data, specifically feature dimension differences and conflicting teacher knowledge.
• To enhance the efficiency and performance of student models in analyzing complex time-series data from wearable sensors.

Main Methods:

• Developed mGLW-KD, integrating a working memory module to unify diverse knowledge into a shared latent workspace.
• Utilized GWT principles to manage attentional control and working memory for prioritizing and retaining key information.
• Applied mGLW-KD to wearable sensor data, enabling efficient knowledge transfer from multiple teacher models to a student model.

Main Results:

• The mGLW-KD framework successfully unified knowledge from multiple teachers despite differing feature dimensions.
• The proposed method demonstrated resilience to noise and signal variations inherent in wearable sensor data.
• The student model achieved superior performance in analyzing time-series data using only time-series input during inference.

Conclusions:

• mGLW-KD effectively overcomes challenges in multimodal KD for wearable sensor applications.
• The integration of topological features and cognitive principles enhances the robustness and efficiency of machine learning models.
• This approach paves the way for more powerful and resource-efficient health monitoring systems using wearable technology.