Optimizing ViT-LoRA: A Memory-Efficient Approach for Fine-Tuning

This paper presents ViT-LoRA, a parameter-efficient Vision Transformer (ViT) adaptation for medical imaging tasks. ViT-LoRA solves the high computational and memory needs of regular ViTs by incorporating Low-Rank Adaptation (LoRA), which reduces the number of trainable parameters from the total to 2.104 million, while keeping the overall parameter size at 137.07 million. As a result, Memory usage reduces significantly from 1568.62 MB to 24.08 MB with ViT-LoRA. In addition, ViT-LoRA also reduces the model size from 1500 MB to 539.2 MB. Nevertheless, ViT-LoRA still outperforms the baseline ViT model with improved testing accuracy of 98.49% compared to 96.60%, and improved F1-score, precision, and recall values. Moreover, the ViT-LoRA model completes training in 400.32 seconds as opposed to 862.23 seconds for the baseline ViT, a 53.5% reduction in training time. Comparative evaluation against ResNet models establishes that ViT-LoRA surpasses both ResNet50 (testing accuracy 85.53%) and ResNet101 (testing accuracy 95.47%) along with less memory utilization, smaller model size, and fewer trainable parameters compared to their baseline equivalents. Experiments with the Lung Infection dataset indicate that ViT-LoRA consistently outperforms. Training was done for all models on an NVIDIA GeForce RTX 4090 GPU with 24 GB RAM.