A General Framework for Efficient Medical Image Analysis via Shared Attention Vision Transformer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Imaging

Background:

• Vision Transformers (ViTs) show potential in medical image analysis but suffer from data and parameter inefficiency.
• Limited ability to capture local features in data-scarce scenarios hinders ViT performance.
• High computational and storage requirements of full fine-tuning limit ViT applicability.

Purpose of the Study:

• To introduce the Shared Attention Vision Transformer (SAViT) for efficient and accurate medical image analysis.
• To address data inefficiency and parameter inefficiency challenges in Vision Transformers for medical applications.
• To develop a model that excels in both training from scratch and transfer learning scenarios.

Main Methods:

• Proposed SAViT model with three modules: Shared Prior Attention (SPA) for data efficiency, MixPool for global modeling, and Low-rank Multi-head Self-Attention (Lr-MSA) for parameter efficiency.
• SPA uses visual prompts to share attention weights across local regions, improving locality and translational invariance.
• MixPool aggregates local features via multi-pooling to maintain long-range dependency, while Lr-MSA reduces computational complexity using low-rank attention weights.

Main Results:

• SAViT demonstrates strong generalization across medical imaging modalities like retinopathy, dermoscopy, and radiography.
• Achieved high data efficiency and superior performance compared to over 20 medical-specific and ViT-based models when trained from scratch.
• Excelled in parameter-efficient transfer learning, outperforming 17 models across 6 datasets with significantly fewer trainable parameters.

Conclusions:

• SAViT offers an efficient and accurate solution for medical image analysis, overcoming key Vision Transformer limitations.
• The model's innovative modules enhance data efficiency, locality, global modeling, and parameter efficiency.
• SAViT presents a promising advancement for developing robust and computationally feasible AI models in healthcare.