SELongVLM: 赋予长视频语言模型自纠正的剪辑选择
概括
通过解决冗余性和改善时空推理,SELongVLM提高了长视频的理解. 这种多式大型语言模型 (MLLM) 框架在各种视频基准上实现了最先进的性能.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 多模式大语言模型 (MLLMs) 在视觉语言推理方面表现有前途.
- 由于复杂的时空依赖和信息冗余,理解长视频具有挑战性.
- 现有的MLLM难以有效处理大量的视频内容,导致性能限制.
研究的目的:
- 开发一个新的框架,SELongVLM,以有效地理解长视频.
- 解决长视频中绝对冗余 (静态内容) 和相对冗余 (无关段).
- 增强MLLM中的时空推理能力,用于复杂的视频分析.
主要方法:
- 介绍了SELongVLM,一个长视频语言模型,有两个协调的分支:剩余令牌修剪器 (RTP) 和语义意识的自我校正选择器 (SCSelector).
- 通过使用框架间剩余建模,RTP通过删除重复的令牌来减轻绝对冗余.
- SCSelector通过查询相关的剪辑选择而减少相对冗余,而无需级注释,以自我纠正机制为指导.
主要成果:
- 在长视频理解的八个基准标准中,SELongVLM显著优于现有模型.
- 在视频MME (65.5%) 和MLVU (69.8%) 等一般基准上取得了最先进的结果.
- 在特殊任务上表现出强的表现,包括精细的时间推理 (TOMATO: 39.2%) 和事件级理解 (EventBench: 69.2%).
结论:
- 在长视频理解中,SELongVLM有效地解决了冗余和弱时空建模的挑战.
- 拟议的框架通过结合动作意识操作和时间记忆来实现强大的时空推理.
- SELongVLM代表了多式联络人工智能的重大进步,用于全面的长视频分析.
相关概念视频
Improving Translational Accuracy
3.7K
3.7K
Improving Translational Accuracy
15.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
15.3K
Fixing Double-strand Breaks
4.6K
4.6K
Fixing Double-strand Breaks
15.9K
The double-stranded structure of DNA has two major advantages. First, it serves as a safe repository of genetic information where one strand serves as the back-up in case the other strand is damaged. Second, the double-helical structure can be wrapped around proteins called histones to form nucleosomes, which can then be tightly wound to form chromosomes. This way, DNA chains up to 2 inches long can be contained within microscopic structures in a cell. A double-stranded break not only damages...
15.9K
Language Development
1.0K
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
1.0K
Detection of Gross Error: The Q Test
7.2K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
7.2K

