Momentor++: Advancing Video Large Language Models With Fine-Grained Long Video Reasoning

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large Language Models (LLMs) excel at text tasks but struggle with video's temporal complexity.
• Existing Video-LLMs lack fine-grained temporal comprehension and efficient segment localization.
• Addressing these limitations is crucial for advancing video understanding.

Purpose of the Study:

• Introduce Momentor, a Video-LLM for fine-grained temporal understanding and localization.
• Develop Moment-10M, a large-scale dataset for training segment-level video instruction tasks.
• Enhance computational efficiency and detail preservation in Video-LLMs.

Main Methods:

• Proposed Momentor, a Video-LLM architecture for detailed temporal video analysis.
• Created Moment-10M dataset using an automatic data generation engine.
• Introduced Spatio-Temporal Token Consolidation (STTC) for parameter-free token merging.

Main Results:

• Momentor demonstrated strong performance in fine-grained temporal understanding and localization tasks.
• Moment-10M dataset facilitated effective training of Video-LLMs.
• Momentor++ with STTC significantly improved computational efficiency for extended videos.

Conclusions:

• Momentor provides robust fine-grained temporal understanding capabilities for videos.
• Momentor++ offers efficient processing of complex, long videos with enhanced temporal context.
• The developed methods advance the field of Video-LLMs for detailed video analysis.