RTF: Recursive TransFusion for Multi-Modal Image Synthesis.

Area of Science:

• Medical Imaging
• Computer Vision
• Artificial Intelligence

Background:

• Multi-modal image synthesis is vital for overcoming real-world imaging limitations.
• CNNs struggle with global representations, causing artifacts in synthesized images.
• Transformers offer global context but require extensive data and parameters.

Purpose of the Study:

• To develop an efficient framework for multi-modal image synthesis.
• To address the limitations of existing CNN and transformer-based methods.
• To improve the accuracy and reduce the computational cost of image synthesis.

Main Methods:

• Proposed a Recursive TransFusion (RTF) framework for multi-modal image synthesis.
• Introduced a TransFusion unit combining CNN-based Local Representation Blocks (LRB) and transformer-based Global Fusion Blocks (GFB) via a Feature Translating Gate (FTG).
• Unfolded the TransFusion unit recursively to progressively extract multi-modal information, reducing network parameters.

Main Results:

• The RTF framework significantly reduces network parameters compared to existing methods.
• RTF achieves superior performance in multi-modal image synthesis.
• Experimental validation on multiple benchmarks confirms the effectiveness of RTF.

Conclusions:

• The proposed RTF framework offers an effective and efficient solution for multi-modal image synthesis.
• RTF overcomes the limitations of traditional methods by integrating local and global feature extraction.
• The recursive design enables parameter reduction without compromising performance, making it suitable for limited data scenarios.