Mstnet: method for glaucoma grading based on multimodal feature fusion of spatial relations

Area of Science:

• Ophthalmology
• Medical Imaging
• Artificial Intelligence

Background:

• Glaucoma, a leading cause of blindness, often results from delayed detection.
• Fundus and Optical Coherence Tomography (OCT) images are crucial for glaucoma diagnosis.
• Existing multimodal models struggle with efficient spatial relationship modeling.

Purpose of the Study:

• To develop an efficient multimodal learning framework for glaucoma classification.
• To enhance spatial relationship modeling between OCT volumes and fundus images.
• To improve early glaucoma detection and management.

Main Methods:

• Utilized a 3D transformer model for feature extraction from OCT volumes.
• Employed downsampling techniques for improved model efficiency.
• Fused features from OCT volumes and fundus images by modeling spatial relationships.

Main Results:

• Achieved remarkable glaucoma grading performance on the GAMMA dataset.
• Outperformed traditional feature fusion methods.
• Demonstrated outstanding classification results through effective spatial relationship modeling.

Conclusions:

• The proposed framework offers a novel approach for glaucoma classification.
• Integrating 3D transformers enhances feature extraction and fusion from multimodal data.
• This work has the potential to significantly improve patient care by aiding early glaucoma detection and treatment.