VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Existing pretraining models primarily focus on vision-language tasks.
• There is a need for models that can jointly understand and generate content across vision, audio, and language modalities.

Purpose of the Study:

• To propose and evaluate the Vision-Audio-Language Omni-perception (VALOR) pretraining model.
• To enable end-to-end multimodal understanding and generation by jointly modeling vision, audio, and language.

Main Methods:

• VALOR utilizes separate encoders for vision, audio, and language, and a decoder for text generation.
• Two pretext tasks, Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC), were designed for pretraining.
• A large-scale dataset, VALOR-1M, comprising 1 million audible videos with annotated captions, was created.

Main Results:

• VALOR effectively learns strong correlations among vision, audio, and language.
• The model demonstrates generalization capabilities across various downstream tasks like retrieval, captioning, and question answering.
• VALOR achieved new state-of-the-art performance on multiple cross-modality benchmarks.

Conclusions:

• VALOR represents a significant advancement in multimodal pretraining research.
• The joint modeling of vision, audio, and language opens new avenues for AI understanding and generation.
• The developed VALOR-1M dataset facilitates further research in tri-modality learning.