Detect and Repair: Robust Self-Supervised Wearable Sensing Under Missing Modalities.

Area of Science:

• Wearable sensor systems
• Human activity recognition
• Multimodal sensor fusion

Background:

• Wearable sensors are vital for monitoring human activities and cognitive states in real-world settings.
• Sensor data degradation or loss due to occlusions, energy limits, or hardware failures poses a significant challenge.
• Robustness in multimodal sensor fusion is crucial for reliable human behavior analysis.

Purpose of the Study:

• Introduce CognifySSL v2.0, a self-supervised learning framework.
• Enable real-time detection and repair of missing sensor modalities.
• Enhance the robustness of wearable sensor systems under adverse conditions.

Main Methods:

• Employed a self-supervised learning approach combining contrastive and masked modeling.
• Utilized a fusion architecture incorporating dropout simulation for multiple sensor signals (IMU, ECG, EDA).
• Simulated real-world missing-modality conditions for comprehensive evaluation.

Main Results:

• Demonstrated effective multimodal detection and reconstruction of missing data on the WESAD dataset.
• Validated unimodal robustness and representation learning capabilities under sensor dropout on the MobiAct dataset.
• Achieved reliable performance in detecting and repairing sensor signal degradation.

Conclusions:

• CognifySSL v2.0 offers a robust solution for handling missing sensor data in wearable systems.
• The framework enhances the reliability of human activity and cognitive state monitoring.
• Open-sourced code and visualization tools facilitate reproducibility and further research in multimodal fusion.