球形视觉转换器用于360$^{\circ }$视频中的视听突出预测
IEEE transactions on pattern analysis and machine intelligence
|August 29, 2025
概括
这项研究引入了在360度视频中预测视觉注意力的新模型, 整合音频线索显著提高了全向视频中突出度预测的准确性.
科学领域:
- 计算机视觉
- 虚拟现实
- 人与计算机的交互
背景情况:
- 全方位视频 (ODV) 提供沉浸式虚拟现实 (VR) 体验,具有完整的视野 (FOV).
- 在360度环境中预测视觉突出性是由于球形扭曲和空间音频的整合而面临的独特挑战.
- 现有的数据集缺乏全面的视听数据来进行360度的突出预测.
研究的目的:
- 通过解决球形扭曲和空间音频集成,将突出度预测扩展到360度视频环境.
- 开发和评估用于ODV的视听突出预测的新型模型.
- 引入一个新的数据集,YT360-EyeTracking,用于培训和评估360°突出预测模型.
主要方法:
- 编辑了YT360-EyeTracking数据集,其中包括81个具有不同视听条件的ODV.
- 建议SalViT360,一个具有球形几何意识的视觉转换器模型用于ODV.
- 开发了SalViT360-AV,该扩展器包含在音频输入条件下的变压器适配器.
主要成果:
- SalViT360和SalViT360-AV在基准数据集上显著优于现有的方法,包括YT360-EyeTracking.
- 证明了将空间音频线索用于增强突出度预测准确性的有效性.
- 验证了模型在复杂的360度场景中预测观众注意力的能力.
结论:
- 整合空间音频对于精确的突出预测至关重要.
- 提出的SalViT360和SalViT360-AV模型代表了360度视觉注意力预测的重大进展.
- YT360-EyeTracking数据集为沉浸式媒体的视听突出性提供了进一步的研究.
相关概念视频
Depth Perception and Spatial Vision
897
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
897
Spherical Coordinates
10.8K
Spherical coordinate systems are preferred over Cartesian, polar, or cylindrical coordinates for systems with spherical symmetry. For example, to describe the surface of a sphere, Cartesian coordinates require all three coordinates. On the other hand, the spherical coordinate system requires only one parameter: the sphere's radius. As a result, the complicated mathematical calculations become simple. Spherical coordinates are used in science and engineering applications like electric and...
10.8K
Relative Motion Analysis using Rotating Axes
530
Consider a component AB undergoing a linear motion. Along with a linear motion, point B also rotates around point A. To comprehend this complex movement, position vectors for both points A and B are established using a stationary reference frame.
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
530
Relative Motion Analysis using Rotating Axes-Problem Solving
448
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
448


