在视觉语言预训练中增强视觉接地,使用位置导向文本提示
IEEE transactions on pattern analysis and machine intelligence
|December 18, 2023
概括
本研究引入了一个位置引导的文本提示符 (PTP),以改善视觉语言预训练 (VLP) 模型的视觉接地. 在交叉模式任务中,PTP增强了对象本地化和推理.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 视觉语言预训练 (VLP) 卓越于对齐图像-文本对交叉模式任务.
- 当前的VLP模型往往缺乏强大的视觉接地和本地化,阻碍了视觉推理等任务的性能.
研究的目的:
- 引入一种新的位置引导文本提示符 (PTP) 范式,以增强 VLP 模型的视觉接地功能.
- 在跨模式学习框架中改进对象本地化和理解.
主要方法:
- PTP范式将图像划分为块,并使用对象检测器来识别每个块内的对象.
- 它将视觉接地重新定义为填空任务,预测对象在块中的位置或对象的位置.
- 整合了二级对象关系,以进一步提高接地性能.
主要成果:
- 通过PTP整合,多个基准的VLP框架得到了显著的改进.
- 值得注意的增长包括Flickr30k检索 (ViLT基线) 的平均回忆率为+5.6和COCO标题的CIDEr (BLIP基线) 的平均回忆率为+5.5.
- 通过在推断过程中丢弃对象探测器,PTP可以获得与基于对象探测器的方法相似的结果,并具有更快的推断速度.
结论:
- 位置导向文本提示符 (PTP) 范式有效地增强了VLP模型中的视觉接地.
- PTP提供了一种计算效率高的方法来改进跨模式学习任务,表现出强大的性能和更快的推断.
更多相关视频
相关概念视频
The Anchoring-and-Adjustment Heuristic
7.2K
In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. However, sometimes, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the...
7.2K
Vision
53.5K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
53.5K
Position Vectors
910
A position vector is a fundamental concept in mathematics that helps determine the position of one point with respect to another point in space. It is a vector that describes the direction and distance between two points. Position vectors are highly useful in the field of math and science, as they help represent spatial relationships and make calculations easier.
For instance, we want to locate a point P(x, y, z) relative to the origin of coordinates O. In that case, we can define a position...
For instance, we want to locate a point P(x, y, z) relative to the origin of coordinates O. In that case, we can define a position...
910


