野生视频:对LMM进行基准测试,以了解视频语言交互.
概括
我们介绍了WildVideo,这是一个用于评估大型多模式模型 (LMM) 的新基准数据集. 这一数据集揭示了当前LMM中显著的幻觉问题,突出了视频语言理解的差距.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 大型多模模式 (LMM) 越来越多地用于视频语言任务.
- 评估LMM的可靠性和准确性,特别是关于幻觉,对于现实世界的应用至关重要.
- 现有的基准可能不能完全捕捉到野外视频理解的复杂性.
研究的目的:
- 介绍WildVideo,这是一个开放世界的基准数据集,用于评估视频语言交互中的LMM幻觉.
- 综合测试感知,认知和上下文理解幻觉.
- 在各种视频数据上提供对单回合和多回合开放式问答 (QA) 的强有力的评估.
主要方法:
- 开发了WildVideo,这是一个开放世界的基准,包含来自第一人称和第三人称视角的1318个视频.
- 定义了9个不同的任务,涵盖多层次感知,认知能力 (常识,世界知识) 和上下文理解 (圆,检索).
- 创建了13704个单回合QA对和1585个多回合对话 (最多5回合).
主要成果:
- 在WildVideo基准上评估了14个常用的LMM.
- 在评估的LMM中确定了重要的幻觉问题.
- 在当前LMM的视频语言理解能力中存在重大差距.
结论:
- 野生视频有效地揭示了LMM中普遍存在的幻觉问题.
- 该基准强调了在处理复杂的视频语言交互时需要改进的LMM.
- 需要进一步的研究来提高LMM在现实世界的场景中的稳定性和准确性.
相关概念视频
Language and Cognition
444
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
444
Language Development
454
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
454
Components of Language
397
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
397


