Regulating Modality Utilization within Multimodal Fusion Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal fusion networks are crucial for aerial imagery analysis but often exhibit modality bias.
• Existing methods struggle to balance the utilization of diverse information sources, limiting performance.

Purpose of the Study:

• To propose a novel modality utilization-based training method for multimodal fusion networks.
• To mitigate modality bias and ensure balanced integration of complementary information streams.

Main Methods:

• Developed a training method to guide network utilization of input modalities.
• Validated the approach on aerial imagery classification and segmentation tasks.
• Assessed the robustness of fusion networks against noise in input modalities.

Main Results:

• The proposed method maintained modality utilization within ±10% of the target.
• Achieved better noise robustness, with a network trained at 75.0% EO utilization showing higher accuracy (81.4%) in noisy conditions compared to traditional methods (73.7%).
• The method maintained an average accuracy of 85.0% across different noise levels, outperforming traditional methods (81.9%).

Conclusions:

• The novel training method effectively balances modality utilization in fusion networks.
• Demonstrated improved performance and significant noise robustness for aerial imagery tasks.
• Presents a significant advancement for multimodal data fusion in diverse applications like robotics, healthcare, and defense.