MoViT: Memorizing Vision Transformers for Medical Image Analysis.

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Computer Vision

Background:

• Transformers and CNNs offer complementary benefits in medical image analysis.
• Transformers require substantial training data, posing challenges in medical imaging due to data limitations.

Purpose of the Study:

• To propose a novel Memorizing Vision Transformer (MoViT) to reduce the reliance on large datasets for training transformer-based medical image analysis models.
• To enhance the efficiency and applicability of transformer architectures in data-scarce medical imaging scenarios.

Main Methods:

• MoViT employs an external memory to cache attention snapshots during training.
• An attention temporal moving average scheme is used to prevent overfitting by updating memories.
• Prototypical attention learning is utilized for inference speedup by distilling memory into smaller subsets.

Main Results:

• MoViT outperforms vanilla transformer models on histology and MRI datasets, particularly with limited annotated data.
• The proposed method achieves competitive performance comparable to Vision Transformer (ViT) using only 3.0% of the training data.
• Demonstrated effectiveness across varied medical image analysis tasks.

Conclusions:

• MoViT serves as a plug-in solution to significantly decrease the training data requirements for transformer architectures in medical image analysis.
• The approach facilitates the development of effective AI models for medical imaging even when data availability is constrained.