Low-Rank Fine-Tuning Meets Cross-modal Analysis: A Robust Framework for Age-Related Macular Degeneration...

Age-related macular degeneration (AMD) is a prevalent retinal degenerative disease among the elderly and is a major cause of irreversible vision loss worldwide. Although color fundus photography (CFP) and optical coherence tomography (OCT) are widely used for AMD diagnosis, information from a single modal is inadequate to fully capture the complex pathological features of AMD. To address this, this study proposes an innovative multi-modal deep learning framework that fine-tunes pre-trained single-modal retinal models for efficient application in multi-modal AMD categorization tasks. Specifically, two independent vision transformer models are used to extract features from CFP and OCT images, followed by deep canonical correlation analysis (DCCA) to perform nonlinear mapping and fusion of features from both modalities, maximizing cross-modal feature correlation. Moreover, to reduce the computational complexity of multi-modal integration, we introduce the low-rank adaptation (LoRA) technique, which uses low-rank decomposition of parameter matrices, achieving superior performance compared to full fine-tuning with only about 0.49% of the trainable parameters. Experimental results on the public dataset MMC-AMD validate the framework's effectiveness. The proposed model achieves an overall F1-score of 0.948, AUC-ROC of 0.991, and accuracy of 0.949, significantly outperforming existing single-modal and multi-modal baseline models, particularly excelling in recognizing complex pathological categories.