Multimodal learning with next-token prediction for large multimodal models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning, integrating text, images, and video, is a key AI challenge.
• Current approaches often rely on specialized architectures like diffusion models or compositional frameworks.
• Next-token prediction has advanced language models but its multimodal application is limited.

Purpose of the Study:

• To introduce Emu3, a novel family of multimodal models.
• To demonstrate a unified approach to multimodal learning using only next-token prediction.
• To achieve state-of-the-art performance across diverse multimodal tasks.

Main Methods:

• Emu3 models were trained exclusively using next-token prediction.
• The models were evaluated on perception and generation tasks across multiple modalities.
• Specific applications included video generation and vision-language-action modeling.

Main Results:

• Emu3 achieved performance comparable to task-specific models and flagship systems.
• The model demonstrated high-fidelity video generation capabilities.
• Emu3 successfully performed interleaved vision-language generation and robotic manipulation tasks.

Conclusions:

• Unified multimodal learning is achievable through next-token prediction.
• Emu3 offers a robust foundation for large-scale multimodal AI.
• This approach paves the way for more general and unified multimodal intelligence.