构建一个开放词汇的视频CLIP模型,使用更好的架构,优化和数据
IEEE transactions on pattern analysis and machine intelligence
|January 23, 2024
概括
开放VCLIP++适应了对比语言图像预训 (CLIP) 进行有效的零拍摄视频识别. 该框架实现了动作识别和视频文本检索任务的最新性能.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 对比语言图像预训 (CLIP) 在零拍摄图像识别方面表现出色,但在视频识别方面应用有限.
- 将CLIP适应视频需要捕捉时空动态,同时保持概括能力.
研究的目的:
- 开发Open-VCLIP++,这是一个增强CLIP的框架,用于强大的零拍摄视频识别.
- 通过使用新的优化技术,解决零历史数据培训的挑战.
- 通过精细的视频描述,改善CLIP的能力向视频领域的转移.
主要方法:
- 开放-VCLIP++ 最小地修改了CLIP,以整合时空视频功能.
- 为有效的训练和测试而没有历史数据,引入了互插体重优化.
- 大型语言模型生成详细的视频描述,与视频功能保持一致,以改进CLIP传输.
主要成果:
- 开放VCLIP++在动作识别数据集上实现了最先进的零射击精度:88.1% (UCF),58.7% (HMDB) 和81.2% (Kinetics-600).
- 该方法的性能远远超过现有方法,高达12.3%.
- 在MSR-VTT数据集上展示了具有竞争力的性能,用于减少微调数据的视频文本检索.
结论:
- 开放VCLIP++为零拍摄视频识别提供了一个高度有效和可通用的框架.
- 建议的交叉重量优化和描述对齐技术对于成功至关重要.
- 这项工作为利用大型预训练模型在视频理解任务中开辟了新的途径.
更多相关视频
相关概念视频
Improving Translational Accuracy
10.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
10.5K
Introduction to Learning
404
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
404
Observational Learning
175
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
175
Extraction: Advanced Methods
447
Metal ions can be separated from one another by complexation with organic ligands–the chelating agent– to form uncharged chelates. Here, the chelating agent must contain hydrophobic groups and behave as a weak acid, losing a proton to bind with the metal. Since most organic ligands used in this process are insoluble or undergo oxidation in the aqueous phase, the chelating agent is initially added to the organic phase and extracted into the aqueous phase. The metal-ligand complex is...
447
Per-Unit Sequence Models
74
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
74
Cognitive Learning
243
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
243


