接触型レーダーおよびウェアラブルIMUセンサーを使用した摂取ジェスチャー検出のための堅牢なマルチモーダル学習フレームワーク

Automated food intake gesture detection is a critical task in the domain of dietary monitoring, with the potential to objectively and continuously record individuals' eating patterns, thereby improving the quality of life (QoL) for diverse populations. Wrist-worn inertial measurement units (IMUs), have been extensively explored for this task and have demonstrated promising performance. Recently, ambient-based contactless radar sensor, has also shown feasibility for intake gesture detection. This study aims to investigate whether the complementary features of wearable and contactless sensors can be effectively leveraged through multimodal learning to further enhance detection performance. Additionally, this study addresses a key challenge in multimodal learning, the reduced robustness when handling missing modalities. To this end, we propose a robust multimodal temporal convolutional network with cross-modal attention (MM-TCN-CMA) framework, designed to efficiently integrate information from IMU and radar sensors, improve detection performance, and maintain effectiveness when modality-incomplete data are encountered during the inference phase. A dataset comprising 52 meal sessions (3,050 eating gestures and 797 drinking gestures) collected from 52 participants is developed and made publicly available for validation. Experimental results demonstrate that the proposed fusion framework achieves a segmental F1-score improvement of 4.3% and 5.2% over unimodal-Radar and unimodal-IMU baselines, respectively. Under missing modality conditions, the framework still yields performance gains of 1.3% and 2.4% in the missing-Radar and missing-IMU scenarios, respectively. To the best of our knowledge, this is the first study to explore a robust multimodal learning framework combining IMU and radar. The proposed robust radar-IMU fusion framework holds potential for broader applications in other continuous, fine-grained human activity recognition (HAR) tasks.