对压缩视频预训练的交叉模式对比面具自动编码器
概括
我们介绍了交叉模式对比面具自动编码器 (C2MAE),这是一种在压缩视频上进行自我监督学习 (SSL) 的新方法. C2MAE通过结合蒙面图像建模和对比学习来提高视频理解来增强表示学习.
科学领域:
- 计算机视觉 计算机视觉
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 自主监督学习 (SSL) 对于利用大型未标记的视频数据集至关重要.
- 压缩视频由于数据稀疏性和噪音,对表示学习提出了独特的挑战.
- 现有的方法往往难以有效地利用来自压缩视频中的不同模式的信息.
研究的目的:
- 提出一种基于变压器的新方法,即交叉模式对比面具自动编码器 (C2MAE),用于压缩视频上的自我监督学习.
- 通过整合掩盖图像建模 (MIM) 和对比学习 (CL) 借口任务来增强表示学习.
- 为了改善压缩视频特征的处理,如I-frame稀疏性和P-frame噪声.
主要方法:
- 采用统一的变压器编码器来处理来自RGB,运动向量和残余的视觉令牌.
- 开发了一种混合SSL框架,将MIM和CL结合起来,将VideoMAE扩展到精细粒度运动感知掩盖 (FGMM) 和多模式重建 (MR).
- 通过比较压缩和原始视频片段的特征,引入了对比的交叉模式学习 (CCL).
主要成果:
- C2MAE显著增强了跨模式交互,有效地弥补了压缩视频数据的局限性.
- 在UCF-101,HMDB-51和Kinetics-400基准上取得了最先进的结果.
- 证明了C2MAE在为视频理解任务提供更强大的预训练模型方面的有效性.
结论:
- C2MAE提供了一个强大而有效的框架,用于在压缩视频上进行自我监督的学习.
- 拟议的FGMM策略和CCL模块有助于高级表示学习.
- 该方法成功地解决了压缩视频数据带来的挑战,从而显著提高了性能.
相关概念视频
Masking and Demasking Agents
2.7K
EDTA titrations may necessitate masking and demasking agents to temporarily protect a particular metal ion in a mixture from the EDTA reaction. These agents facilitate the sequential analysis of the metal ions by forming stable complexes with some—but not all—metal ions during certain steps.
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
2.7K
Deconvolution
260
Deconvolution, also known as inverse filtering, is the process of extracting the impulse response from known input and output signals. This technique is vital in scenarios where the system's characteristics are unknown, and they must be inferred from the observable signals.
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
260
Force Classification
1.6K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.6K
Associative Learning
593
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
593
Sign Test for Matched Pairs
212
The sign test for matched pairs offers a robust method for comparing two paired samples, often for the effects of an intervention in one of them. This method is very useful in situations where the underlying distribution of the data is unknown. The test compares two related samples—often pre- and post-treatment measurements on the same subjects—to determine if there are significant differences in their median values.
To conduct the sign test, we first calculate the differences in...
To conduct the sign test, we first calculate the differences in...
212
Prosopagnosia
260
Prosopagnosia, also known as face blindness, is the inability to recognize faces. In severe cases, individuals with prosopagnosia may not recognize close family members, including parents and spouses, by their faces. For instance, someone with prosopagnosia might walk past their child in a crowd, only realizing their mistake upon noticing their child's distinctive backpack or favorite jacket. Prosopagnosia specifically impairs facial recognition, while the recognition of other objects or...
260


