跨模态遥感通过上下文和不确定性意识提示符检索图像-文本
IEEE transactions on neural networks and learning systems
|September 24, 2024
概括
本研究介绍了用于跨模式遥感图像文本检索的上下文和不确定性感应提示 (CUP) 网络. CUP有效地将大型模型调整为遥感数据,以更少的计算资源提高检索性能.
科学领域:
- 遥感 遥感 遥感 遥感
- 计算机视觉 计算机视觉
- 人工智能的人工智能
背景情况:
- 跨模态遥感图像文本检索 (CMRSITR) 是一个不断发展的领域,利用大型预训练模型.
- 现有方法面临的挑战是微调的高计算成本和自然和遥感图像之间的域差距.
- 这些局限性阻碍了强大的图像-文本模型在遥感环境中的有效应用.
研究的目的:
- 提出一个新的CMRSITR网络,即上下文和不确定性意识的提示 (CUP),解决计算和域调整挑战.
- 通过使用即时调,使大型预训练模型的高效知识转移到远程传感任务.
- 为了提高模型对遥感图像特征的理解,并减轻不确定性,以提高检索准确度.
主要方法:
- 实现了提示调整,通过只训练提示令牌来降低计算负担.
- 开发了一个提示生成模块 (PGM) 来创建远程传感特定的提示令牌,弥合了与自然图像预训练的差距.
- 集成了一个不确定性估计模块 (UEM) 来解决语义 misalignment 和数据不确定性.
主要成果:
- 拟议的CUP网络在CMRSITR的三个基准数据集上取得了竞争性表现.
- 与完整模型微调相比,快速调整显著降低了对大量计算资源的需求.
- 面向RS的提示和不确定性估计有效地提高了模型解释遥感图像的能力.
结论:
- CUP提供了一种高效有效的解决方案,用于跨模态遥感图像文本检索.
- 该方法成功地使大规模模型适应遥感数据的独特特征.
- 这种方法显示了推进遥感信息提取和分析的巨大潜力.
更多相关视频
05:58Using Rapid Serial Visual Presentation to Measure Set-Specific Capture, a Consequence of Distraction While Multitasking
Published on: August 29, 2018
8.8K
07:34Perceptual and Category Processing of the Uncanny Valley Hypothesis' Dimension of Human Likeness: Some Methodological Issues
Published on: June 3, 2013
17.3K
相关概念视频
Depth Perception and Spatial Vision
605
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
605
Imaging Biological Samples with Optical Microscopy
4.6K
Optical microscopy uses optic principles to provide detailed images of samples. Antonie van Leeuwenhoek designed the first compound optical microscope in the 17th century to visualize blood cells, bacteria, and yeast cells. In 1830, Joseph Jackson Lister created an essentially modern light microscope. The 20th century saw the development of microscopes with enhanced magnification and resolution.
In optical microscopy, the specimen to be viewed is placed on a glass slide and clipped on the stage...
In optical microscopy, the specimen to be viewed is placed on a glass slide and clipped on the stage...
4.6K
