WildVideo: Benchmarking LMMs for Understanding Video-Language Interaction.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Large Multi-modal Models (LMMs) are increasingly used for video-language tasks.
• Assessing the reliability and accuracy of LMMs, particularly regarding hallucinations, is crucial for real-world applications.
• Existing benchmarks may not fully capture the complexities of in-the-wild video understanding.

Purpose of the Study:

• Introduce WildVideo, an open-world benchmark dataset to evaluate LMM hallucination in video-language interaction.
• Comprehensively test perceptual, cognitive, and contextual comprehension hallucination.
• Provide a robust evaluation for single-turn and multi-turn open-ended question-answering (QA) on diverse video data.

Main Methods:

• Developed WildVideo, an open-world benchmark with 1,318 videos from first-person and third-person perspectives.
• Defined 9 distinct tasks covering multi-level perception, cognitive abilities (commonsense, world knowledge), and contextual comprehension (ellipsis, retrieval).
• Created 13,704 single-turn QA pairs and 1,585 multi-turn dialogues (up to 5 turns).

Main Results:

• Evaluated 14 commonly-used LMMs on the WildVideo benchmark.
• Identified significant hallucination issues across evaluated LMMs.
• Demonstrated substantial gaps in current LMMs' video-language understanding capabilities.

Conclusions:

• WildVideo effectively reveals prevalent hallucination problems in LMMs.
• The benchmark highlights the need for improved LMMs in handling complex video-language interactions.
• Further research is required to enhance the robustness and accuracy of LMMs in real-world scenarios.