对传感器获取的视频进行文本导向视觉表示优化 时间接地
Yun Tian1, Xiaobo Guo1, Jinsong Wang1
1School of Optoelectronic Engineering, Changchun University of Science and Technology, Changchun 130022, China.
Sensors (Basel, Switzerland)
|August 14, 2025
概括
本研究引入了一种新的框架,通过使用文本指导优化视觉表示来改善视频时间接地 (VTG). 这种方法有效地弥合了跨模式的差距,增强了语义对齐,用于准确的视频段本地化.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 视频时间接地 (VTG) 旨在识别与文本查询相匹配的视频中的特定时间段.
- 现有的方法因多余的视觉数据和独立的文本/视频处理而扎于交叉模式的语义 misalignment.
- 这种错位阻碍了基于自然语言描述的相关视频内容的准确本地化.
研究的目的:
- 提出一个以文本为导向的视觉表示优化框架,以增强视频信号中的语义解释.
- 通过利用文本信息来缩小跨模式差距,专注于相关的时空视频内容.
- 通过完善视觉表示来提高视频时间接地的准确性.
主要方法:
- 利用CLIP的统一的交叉模式嵌入空间进行表示结构化.
- 引入了一个空间视觉表示优化 (SVRO) 模块,通过选择突出补丁来完善框架内空间信息.
- 开发了一个时间视觉表示优化 (TVRO) 模块,具有时间三重损失,以完善跨时间关系和剪辑语义.
- 纳入自我监督的对比损失,以改善剪贴间的歧视.
主要成果:
- 拟议的框架在广泛使用的基准数据集上表现出卓越的表现:Charades-STA,ActivityNet标题和TACoS.
- 在多个评估指标中表现优于现有的最先进方法.
- 有效地增强了文本查询和视频内容之间的语义对齐.
结论:
- 文本导向的视觉表示优化框架显著改善了视频的时间接地.
- SVRO和TVRO模块有效地解决了分别的空间和时间表示挑战.
- 该方法为解决视频理解任务中的交叉模式语义错位提供了一个有希望的方向.
相关概念视频
Relative Motion Analysis using Rotating Axes-Problem Solving
449
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
449
Depth Perception and Spatial Vision
909
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
909
Relative Motion Analysis using Rotating Axes
533
Consider a component AB undergoing a linear motion. Along with a linear motion, point B also rotates around point A. To comprehend this complex movement, position vectors for both points A and B are established using a stationary reference frame.
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
533


