VIDHALLUC: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Multimodal large language models (MLLMs) demonstrate advanced video understanding capabilities.
• Hallucination, the generation of inaccurate content, is a significant challenge in MLLM video analysis.
• Existing MLLM visual encoders often confuse visually similar but semantically distinct video content.

Purpose of the Study:

• To introduce VIDHALLUC, the largest benchmark for evaluating MLLM hallucinations in video understanding.
• To assess MLLM vulnerability to hallucinations across action, temporal sequence, and scene transition.
• To propose DINO-HEAL, a novel method to mitigate video hallucinations.

Main Methods:

• Development of the VIDHALLUC benchmark comprising 5,002 paired videos.
• Comprehensive testing of various MLLMs on the VIDHALLUC benchmark.
• Introduction of DINO-HEAL, a training-free technique using DINOv2 spatial saliency for feature reweighting.

Main Results:

• Most MLLMs exhibit significant hallucinations across action, temporal sequence, and scene transition dimensions.
• DINO-HEAL effectively reduces hallucinations in MLLMs.
• DINO-HEAL achieved an average improvement of 3.02% in mitigating hallucinations across tasks.

Conclusions:

• MLLMs are susceptible to hallucinations in video understanding tasks.
• The VIDHALLUC benchmark provides a critical resource for assessing and addressing these limitations.
• DINO-HEAL offers a practical and effective solution for reducing MLLM video hallucinations without retraining.