Learning Two-Stream CNN for Multi-Modal Age-Related Macular Degeneration Categorization.

Area of Science:

• Ophthalmology and Medical Imaging
• Artificial Intelligence in Healthcare
• Computational Pathology

Background:

• Age-related Macular Degeneration (AMD) is a prevalent condition affecting individuals over 50.
• Existing AMD categorization methods primarily rely on single imaging modalities (CFP or OCT).
• Multi-modal approaches for AMD diagnosis are clinically relevant but underexplored.

Purpose of the Study:

• To develop an automated system for Age-related Macular Degeneration (AMD) categorization using multi-modal imaging.
• To propose an end-to-end deep learning framework for integrating Color Fundus Photographs (CFP) and OCT B-scan images.
• To enhance the interpretability and training efficiency of multi-modal deep learning models for AMD.

Main Methods:

• Implementation of a two-stream Convolutional Neural Network (CNN) for processing CFP and OCT data.
• Spatially-invariant fusion technique to integrate information from both imaging modalities.
• Extension of Class Activation Mapping (CAM) for multi-modal interpretability and development of GAN-based data augmentation and Loose Pairing for improved training.

Main Results:

• The proposed multi-modal CNN (MM-CNN) demonstrated superior performance compared to baseline methods.
• The model achieved higher F1 scores and accuracy in categorizing AMD using combined CFP and OCT data.
• Visual interpretability was enhanced through multi-modal CAM, aiding in understanding modality contributions.

Conclusions:

• End-to-end multi-modal deep learning offers a promising avenue for accurate AMD categorization.
• Integrating CFP and OCT data significantly improves diagnostic performance over single-modality approaches.
• The developed MM-CNN framework, coupled with novel augmentation and interpretability techniques, provides a robust solution for clinical application.