CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal Large Language Models (MLLMs) leverage implicit knowledge for cross-modal learning.
• Advances in audio-visual question answering (AVQA) tasks are hindered by audio-visual ambiguity and hallucination in existing MLLMs.

Purpose of the Study:

• To enhance MLLMs for robust audio-visual understanding and accurate response generation.
• To address challenges of ambiguity and hallucination in MLLMs for AVQA tasks.

Main Methods:

• Introduction of the Sequential Question-guided Module (SQM) for improved audio-visual grounding.
• Implementation of Ambiguity Scoring Direct Preference Optimization (AS-DPO) to mitigate biased descriptions.
• Development of the Audio-visual Hallucination Benchmark (AVHbench) to evaluate MLLM hallucination deficits.

Main Results:

• CAT+ demonstrates superior performance in video-based understanding and AVQA tasks.
• The SQM module ensures robust audio-visual grounding.
• AS-DPO effectively corrects biases toward ambiguous descriptions, and AVHbench provides a new standard for evaluating hallucinations.

Conclusions:

• The proposed CAT+ method significantly improves MLLM performance in AVQA by tackling ambiguity and hallucination.
• The developed AVHbench is a valuable resource for assessing and advancing MLLMs in dynamic audio-visual scenarios.