SELongVLM: Empowering Long Video Language Models With Self-Corrective Clip Selection

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs) show promise in visual-language reasoning.
• Understanding long videos is challenging due to complex spatiotemporal dependencies and information redundancy.
• Existing MLLMs struggle to effectively process extensive video content, leading to performance limitations.

Purpose of the Study:

• To develop a novel framework, SELongVLM, for effective long-video understanding.
• To address absolute redundancy (static content) and relative redundancy (irrelevant segments) in long videos.
• To enhance spatiotemporal reasoning capabilities in MLLMs for complex video analysis.

Main Methods:

• Introduced SELongVLM, a long video language model with two coordinated branches: Residual Token Pruner (RTP) and Semantic-aware Self-Correction Selector (SCSelector).
• RTP mitigates absolute redundancy by removing repetitive tokens using inter-frame residual modeling.
• SCSelector reduces relative redundancy through query-relevant clip selection without frame-level annotations, guided by a self-correcting mechanism.

Main Results:

• SELongVLM significantly outperforms existing models across eight benchmarks for long-video understanding.
• Achieved state-of-the-art results on general benchmarks like VideoMME (65.5%) and MLVU (69.8%).
• Demonstrated strong performance on specialized tasks, including fine-grained temporal reasoning (TOMATO: 39.2%) and event-level understanding (EventBench: 69.2%).

Conclusions:

• SELongVLM effectively tackles the challenges of redundancy and weak spatiotemporal modeling in long-video understanding.
• The proposed framework enables robust spatiotemporal inference by incorporating action-aware operations and temporal memory.
• SELongVLM represents a significant advancement in multimodal AI for comprehensive long-video analysis.