A unified multimodal classification framework based on deep metric learning.

Area of Science:

• Multimodal Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Multimodal classification algorithms are crucial for analyzing data from various sources.
• Existing methods often focus on specific tasks and data types, limiting their applicability.
• There is a need for a versatile framework capable of handling diverse multimodal classification challenges.

Purpose of the Study:

• To introduce a unified multimodal classification framework (UMCF) adaptable to various tasks and data modalities.
• To develop a task-independent framework with interchangeable unimodal feature extraction modules.
• To enhance the extraction of latent features within multimodal data through deep metric learning.

Main Methods:

• Proposed a unified multimodal classification framework (UMCF) that is task-independent.
• Implemented an adaptive unimodal feature extraction module for diverse data types.
• Utilized deep metric learning, including metric-based triplet learning and contrastive pairwise learning, to capture intra- and inter-modal relationships.

Main Results:

• UMCF demonstrated superior performance in multimodal classification tasks, including fake news detection and sentiment analysis.
• The framework effectively extracts features from multimodal data.
• Achieved an average F1 score improvement of 2.3% over the best fake news detection baselines.

Conclusions:

• The proposed UMCF offers a flexible and effective solution for diverse multimodal classification problems.
• The task-independent design and adaptive feature extraction enhance its generalizability.
• UMCF significantly advances the state-of-the-art in multimodal machine learning performance.