SMURF: Statistical Modality Uniqueness and Redundancy Factorization.

Area of Science:

• Multimodal machine learning
• Affective computing
• Artificial intelligence

Background:

• Multimodal late fusion models combine information from different sources (e.g., vision, audio, text) for improved predictions.
• Current late fusion methods lack interpretability regarding individual modality contributions.
• There is a need to enhance the robustness of these models to missing data modalities.

Purpose of the Study:

• To improve the interpretability of late fusion models by factorizing modality contributions.
• To enhance the robustness of late fusion models to missing modalities.
• To introduce a novel late fusion method, Statistical Modality Uniqueness and Redundancy Factorization (SMURF).

Main Methods:

• Proposing SMURF, a late fusion method that factorizes contributions into unique and pairwise redundant components.
• Unique contributions are uncorrelated with other modalities.
• Pairwise redundant contributions are maximally correlated between two modalities.

Main Results:

• SMURF's factorization was verified on synthetic data and showed no degradation in predictive performance across eight affective datasets.
• The factorization learned by SMURF correlated significantly with human judgments on three datasets.
• SMURF demonstrated improved robustness to missing modalities compared to baseline methods.

Conclusions:

• SMURF successfully factorizes modality contributions, enhancing interpretability in late fusion.
• The proposed method improves model robustness when modalities are missing.
• SMURF offers a promising approach for developing more transparent and resilient multimodal AI systems.