Multimodal Learning With Transformers: A Survey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Transformers are powerful neural network learners excelling in various machine learning tasks.
• Multimodal learning, leveraging diverse data types, is a growing area in AI research.
• The proliferation of Big Data fuels advancements in Transformer-based multimodal applications.

Purpose of the Study:

• To provide a comprehensive survey of Transformer techniques specifically designed for multimodal data.
• To systematically review different Transformer architectures, including Vanilla Transformer, Vision Transformer, and multimodal Transformers.
• To examine the application paradigms of multimodal Transformers, focusing on pretraining and specific task-oriented approaches.

Main Methods:

• A geometrically topological perspective is used to review Transformer architectures.
• Applications are reviewed through the lenses of multimodal pretraining and specific multimodal tasks.
• Common challenges and design choices in multimodal Transformer models and applications are summarized.

Main Results:

• The survey systematically categorizes and reviews various Transformer models for multimodal data.
• Key application paradigms, including pretraining and task-specific uses, are analyzed.
• Common challenges and design patterns across multimodal Transformer research are identified.

Conclusions:

• Transformer-based multimodal learning is a significant and rapidly evolving field in AI.
• Understanding the reviewed architectures, applications, and challenges is crucial for future research.
• Open problems and potential research directions are discussed to guide the community.