Multimodal Multiresolution Data Fusion Using Convolutional Neural Networks for IoT Wearable Sensing.

Area of Science:

• Biomedical Engineering
• Artificial Intelligence
• Signal Processing

Background:

• Internet of Things (IoT) sensors enable simultaneous data acquisition for enhanced inferences.
• Multimodal and multiresolution sensor data present challenges for fusion due to differing frequency resolutions.

Purpose of the Study:

• To propose a novel convolutional neural network (CNN) model for fusing multimodal and multiresolution sensor data.
• To enable temporal inferences, such as high-resolution event detection, without data resampling.
• To evaluate the model's performance in sleep apnea event detection.

Main Methods:

• Developed a novel CNN model for multimodal and multiresolution sensor data fusion.
• Applied a selective dropout technique to prevent overfitting and enhance fusion robustness.
• Evaluated the model using electrocardiogram (ECG), Peripheral oxygen saturation (SpO2), and abdominal movement signals from a sleep apnea database.

Main Results:

• The fusion model achieved 99.72% accuracy and 98.98% sensitivity for sleep apnea detection.
• Performance improved incrementally with the number of fused sensors, demonstrating generalizability.
• Energy consumption was estimated at 5.61 μJ per classification for on-chip implementation.
• Feasibility of pruning for complexity reduction was investigated.

Conclusions:

• The proposed CNN model effectively fuses multimodal and multiresolution sensor data for accurate sleep apnea detection.
• The model demonstrates generalizability and robustness, with potential for low-power on-chip implementation.
• Selective dropout and pruning offer strategies for optimizing model performance and complexity.