Redefining lightweight vision models for healthcare AI

Area of Science:

• Artificial Intelligence
• Computer Vision
• Medical Imaging

Background:

• Traditional medical vision models are parameter-heavy, prompting research into architectural efficiency.
• Achieving high classification performance without sacrificing efficiency is a key challenge.

Purpose of the Study:

• Introduce MedLiT-seed (2.1M parameters) and MedLiT-nano (0.75M parameters), ultra-lightweight vision transformers.
• Enable efficient and scalable medical image analysis.

Main Methods:

• Utilized a streamlined Mixture-of-Experts (MoE) architecture with SwiGLU, grouped query attention, and depth-wise scaling.
• Pre-trained models using masked autoencoding on ImageNet and MedMNIST, followed by fine-tuning on 12 MedMNIST 2D subsets.
• Compared performance against benchmark models like ResNet, MedViT, and AutoML systems.

Main Results:

• MedLiT-seed achieved top AUC on 4 subsets and strong performance on others, outperforming models 10-20x larger.
• MedLiT-nano matched or exceeded ResNet-18 and AutoML baselines on several subsets.
• Transfer learning from ImageNet improved convergence and generalization; increasing embedding size was more impactful than increasing expert count.

Conclusions:

• MedLiT's MoE-based token routing offers a viable pathway for competitive accuracy with minimal parameters (around 2M).
• Selective computation routing through specialized experts is an effective design for compact medical vision models.
• This architecture is suitable for low-resource settings and scalable fine-tuning, though multi-label task limitations require future refinement.