Progressive Learning of a Multimodal Classifier Accounting for Different Modality Combinations.

Area of Science:

• Computer Science, Machine Learning
• Artificial Intelligence

Background:

• Multimodal classification models typically require all modalities for accurate predictions.
• Existing models lack portability to subsets of modalities and suffer performance degradation with missing data.

Purpose of the Study:

• To address the multimodal portability and missing modality problems in classification tasks.
• To develop a deep learning model and learning strategy for robust and portable multimodal classification.

Main Methods:

• Proposed KModNet, a transformer-based deep learning model with branches for k-combinations of modalities.
• Introduced progressive learning, a multi-step framework training k-modal models sequentially.
• Implemented random ablation of training data to simulate missing modalities.

Main Results:

• KModNet and progressive learning effectively addressed both missing modality and portability issues.
• The framework demonstrated enhanced robustness in audio-video-thermal person and audio-video emotion classification.
• Validated on Speaking Faces, RAVDESS, and SAVEE datasets, showing improved performance with missing data.

Conclusions:

• The proposed progressive learning framework significantly enhances the robustness of multimodal classification.
• The KModNet model demonstrates portability, enabling classification with various subsets of modalities.
• This approach offers a more flexible and reliable solution for real-world multimodal classification scenarios.