Multi-modal classification of retinal disease based on convolutional neural network

Retinal diseases such as age-related macular degeneration and diabetic retinopathy will lead to irreversible blindness without timely diagnosis and treatment. Optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA) images provide complementary views of the retina, and the integration of the two imaging modalities can improve the accuracy of retinal disease classification. We propose a multi-modal classification model consisting of two branches to automatically diagnose retinal diseases, in which OCT and OCTA images are efficiently integrated to improve both the accuracy and efficiency of disease diagnosis. A bright line cropping is used to remove the useless black edge region while preserving the lesion features and reducing the calculation load. To solve the insufficient data issue, data enhancement and loose matching methods are adopted to increase the data amount. A two-step training method is used to train our proposed model, alleviating the limited training images. Our model is tested on an external test set instead of a training set, making the classification results more rigorous. The intermediate fusion and two-step training methods are adopted in our multiple classification model, achieving 0.9667, 0.9418, 0.8569, 0.9422, and 0.8921 in average accuracy, precision, recall, specificity, and F1-Score, respectively. Our multi-modal model outperforms the single-modal model, the early, and late fusion multi-modal model in accuracy. Our model offers doctors less human error, lower cost, more uniform, and effective mass screening, thus providing a solution to improve deep learning performance in terms of a relatively fewer number of training data and even more imbalanced classes.