Deep Learning-Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

Area of Science:

• Biomedical Engineering
• Computer Science
• Human-Computer Interaction

Background:

• Multimodal wearable technologies offer potential for personalized monitoring of human activities, including eating habits.
• A key challenge is selecting discriminative information from high-dimensional data for computationally constrained environments.
• Existing fusion algorithms are often too complex for low-level integration at the data source.

Purpose of the Study:

• To develop a computationally efficient data fusion technique for multimodal wearable sensors.
• To achieve comprehensive human activity recognition in a lower-dimensional space.
• To explore statistical dependencies and intermodality correlations in multisensory data for activity patterns.

Main Methods:

• Information from multiple sensor sources is transformed into a 2D space, regardless of the number of sources.
• A hypothesis-driven approach utilizes the covariance matrix of sensor signals, encoded as a contour representation.
• These 2D representations serve as input for a deep learning model to identify activity-specific patterns.

Main Results:

• The proposed fusion algorithm was validated across two distinct scenarios with varying parameters (e.g., sensors, activities, subjects).
• A precision of 0.803 was achieved in a complex daily living activity recognition scenario using leave-one-subject-out cross-validation.
• The impact of missing data on the performance degradation was also assessed.

Conclusions:

• The developed fusion technique effectively embeds joint variability from different modalities into a single 2D representation.
• This approach provides a global view of daily human activities while maintaining high performance in recognition tasks.
• The method addresses the need for efficient, low-level fusion in wearable technology for activity recognition.