Robust Multimodal Learning Framework for Intake Gesture Detection Using Contactless Radar and Wearable IMU Sensors

Area of Science:

• Human-Computer Interaction
• Biomedical Engineering
• Machine Learning

Background:

• Automated food intake gesture detection is crucial for objective dietary monitoring and improving quality of life.
• Wrist-worn inertial measurement units (IMUs) and contactless radar sensors show promise for detecting eating patterns.
• Multimodal learning offers potential to enhance detection performance by combining sensor data.

Purpose of the Study:

• To investigate the synergistic benefits of combining wearable IMU and contactless radar sensors for food intake gesture detection.
• To develop a robust multimodal learning framework capable of handling missing sensor data.
• To improve the accuracy and reliability of dietary monitoring systems.

Main Methods:

• Proposed a robust multimodal temporal convolutional network with cross-modal attention (MM-TCN-CMA) framework.
• Integrated data from IMU and radar sensors for multimodal learning.
• Developed and validated a dataset of 52 meal sessions (3,050 eating, 797 drinking gestures) from 52 participants.

Main Results:

• The MM-TCN-CMA framework achieved a segmental F1-score improvement of 4.3% (vs. unimodal radar) and 5.2% (vs. unimodal IMU).
• The framework demonstrated robustness under missing modality conditions, showing performance gains of 1.3% (missing radar) and 2.4% (missing IMU).
• This is the first study to explore a robust multimodal learning framework combining IMU and radar for this task.

Conclusions:

• The proposed radar-IMU fusion framework effectively leverages complementary sensor features for enhanced food intake gesture detection.
• The MM-TCN-CMA framework offers improved robustness and performance, particularly in scenarios with incomplete sensor data.
• This multimodal approach has potential for broader applications in continuous, fine-grained human activity recognition.