具有重量转换的双流特征分解网络,用于多模式图像融合
Tianqing Hu1, Xiaofei Nan1, Xiabing Zhou2
1School of Computer Science and Artificial Intelligence, Zhengzhou University, Zhengzhou, 450001, China.
Scientific reports
|March 3, 2025
概括
这项研究介绍了一种新的多模式图像融合模型,将变压器和CNN结合起来,以增强红外和医疗成像. 该方法有效地融合特征,保留细节,以改进对象检测和视觉任务.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 图像处理 图像处理
背景情况:
- 多模式图像融合旨在将来自不同来源的信息整合到一个单一的图像中.
- 使用卷积神经网络 (CNN) 的现有方法具有有限的受体场,而基于变压器的方法是计算密集的.
- 当前的方法往往无法充分探索跨领域的信息,并且无法处理跨不同频率的特征提取.
研究的目的:
- 为多模式图像提出创新的图像融合模型,包括红外/可见和医疗成像对.
- 为了利用变压器和CNN的互补优势,有效地建模各种特征类型和学习范围.
- 通过生成具有保留突出信息和互补特征的融合图像来增强下游视觉任务.
主要方法:
- 一个共享的基于变压器的编码器,用于远程学习,结合了模式内,模式间和特征对齐块.
- 一个基于CNN的私人双流编码器,用于低频和高频特征提取,具有双域选择机制和可逆神经网络.
- 一个基于交叉注意力的Swin变压器块,嵌入了权重转换,用于高效的跨域信息探索和具有动态权重的统一损失函数.
主要成果:
- 拟议的模型有效地保留了融合图像中的热目标和背景纹理细节.
- 定性和定量分析表明,与最先进的图像融合方法相比,性能优越.
- 在随后的视觉任务,如对象检测等的执行方面,观察到显著的改进.
结论:
- 开发的多模态图像融合模型使用混合变压器-CNN架构成功集成了来自不同来源的功能.
- 该模型通过增强特征提取,跨领域信息探索和效率来解决现有方法的局限性.
- 拟议的方法为高质量的图像融合提供了一个强大的解决方案,有利于各种下游计算机视觉应用.
相关概念视频
01:20Deconvolution
127
Deconvolution, also known as inverse filtering, is the process of extracting the impulse response from known input and output signals. This technique is vital in scenarios where the system's characteristics are unknown, and they must be inferred from the observable signals.
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
127
01:22Multi-input and Multi-variable systems
93
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
93
00:57Weighted Mean
4.9K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
4.9K
01:17Convolution Properties II
166
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
166
01:20Convolution Properties I
131
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
131
01:18Reducing Line Loss
141
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
141