MAT: Mixing Attention Transfer From Multiple Transformers for Medical Tasks

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Imaging

Background:

• Transformers are powerful for image analysis but require large datasets.
• Medical AI often faces challenges with limited data availability.

Purpose of the Study:

• To introduce a novel multi-source transfer learning approach for transformers in medical imaging.
• To address the challenge of limited data in medical AI tasks.

Main Methods:

• Proposed Mixing Attention Transfer (MAT), a method designed for transformers.
• MAT utilizes a Mixing Attention layer with token-level Routing and Fusion, and sequence-level Aligned-Attention.
• It enables knowledge transfer from multiple source transformers to target medical tasks.

Main Results:

• Demonstrated the effectiveness of MAT across three medical scenarios: noisy-labeled, class-imbalanced, and fine-grained tasks.
• MAT successfully harnesses and merges knowledge from multiple sources at token and layer levels.
• Achieved improved performance on target medical tasks with limited data.

Conclusions:

• MAT is the first multi-source transfer learning approach specifically for transformers in medical AI.
• The proposed method enhances transformer performance in data-scarce medical imaging applications.
• MAT offers a viable solution for improving AI in challenging medical scenarios.