MultiBench: Multiscale Benchmarks for Multimodal Representation Learning.

Area of Science:

• Multimodal Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Multimodal representation learning integrates diverse data sources, crucial for applications in multimedia, healthcare, and robotics.
• Existing research faces limitations in evaluating generalization, complexity, and robustness to noisy or missing data.
• Limited resources hinder progress in understudied modalities and tasks.

Purpose of the Study:

• To introduce MultiBench, a large-scale, unified benchmark for systematic multimodal learning research.
• To accelerate progress by providing standardized tools for evaluating generalization, complexity, and robustness.
• To address challenges in scalability and real-world data imperfections.

Main Methods:

• Developed an automated end-to-end machine learning pipeline for data loading, setup, and evaluation.
• Created a comprehensive methodology to assess generalization, time/space complexity, and modality robustness.
• Provided standardized implementations of 20 core multimodal learning approaches.

Main Results:

• MultiBench spans 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
• Standardized implementations improved state-of-the-art performance on 9 out of 15 datasets.
• Demonstrated the effectiveness of cross-domain method application.

Conclusions:

• MultiBench unifies disjointed efforts in multimodal machine learning, enhancing ease of use, accessibility, and reproducibility.
• It provides a clear path for understanding multimodal model capabilities and limitations.
• The benchmark and implementations are publicly available and will be regularly updated.