SarAdapter: Prioritizing Attention on Semantic-Aware Representative Tokens for Enhanced Medical Image Segmentation.

Area of Science:

• Medical Image Analysis
• Computer Vision
• Artificial Intelligence

Background:

• Transformer networks show promise for medical image segmentation but suffer from high computational costs.
• Existing acceleration methods have limitations, hindering optimal quality-complexity trade-offs.

Purpose of the Study:

• To develop an efficient transformer-based segmentation method for medical imaging.
• To improve the balance between accuracy, model size, and computational efficiency.

Main Methods:

• Proposed the semantic-aware adapter (SarAdapter) integrating semantic-based routing with neural operators (ViT and CNN).
• Implemented a strategy to merge similar tokens into low-resolution regions and preserve distinct tokens in high-resolution regions.
• Introduced a Mixed-adapter unit for adaptive selection of convolutional operators at different scales.

Main Results:

• Achieved a superior balance between accuracy, model size, and efficiency across four medical datasets and three modalities.
• Demonstrated state-of-the-art segmentation quality on the Synapse dataset.
• Reduced the number of tokens by 65.6%, significantly enhancing the efficiency of Vision Transformers (ViTs) for segmentation.

Conclusions:

• The SarAdapter effectively addresses the quality-complexity trade-off in transformer-based medical image segmentation.
• This method offers a substantial improvement in efficiency for ViTs in segmentation tasks, making them more applicable in resource-constrained environments.