MTID-TS: Multimodal Training with Incomplete Data using Teacher-Student-Based Strategy on Medical Domain

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Multimodal learning integrates diverse data but struggles with missing modalities, impacting reliability in real-world AI applications.
• Medical and human activity recognition domains are particularly susceptible to performance degradation due to incomplete data.

Purpose of the Study:

• To develop a robust framework for multimodal learning that effectively handles uncertain missing modality conditions.
• To enhance the reliability and performance of AI models in applications with incomplete data.

Main Methods:

• Proposed a novel framework, Multimodal Training with Incomplete Data using a Teacher-Student-based Strategy (MTID-TS).
• Utilized knowledge distillation from teacher models to a student model, transferring probability outputs for performance boosting.
• Introduced a Missing Indicator Vector (MIV) to guide model training and inference based on modality availability.
• Integrated a stacking ensemble approach and GradNorm for improved classification accuracy and balanced loss contributions.

Main Results:

• The MTID-TS framework demonstrated improved classification performance and robustness on multimodal datasets.
• The proposed approach outperformed existing methods in handling uncertain missing modality problems.
• Experimental validation was conducted on real-world datasets from the medical and human activity recognition domains.

Conclusions:

• The MTID-TS framework offers a robust solution for training and inference in multimodal learning systems facing incomplete data.
• The approach effectively adapts multimodal AI to the complexities of real-world applications, enhancing model reliability.