MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning.

Area of Science:

• Artificial Intelligence
• Neuroscience
• Medical Imaging

Background:

• Prompt learning effectively fine-tunes multimodal models for various tasks.
• Current prompt learning for neurological disorder diagnosis has limitations, including treating all image patches equally and ignoring crucial brain network structural information.

Purpose of the Study:

• To develop a novel prompt learning model for neurological disorder diagnosis that addresses the limitations of existing methods.
• To enhance the accuracy and interpretability of multimodal model fine-tuning for neurological conditions.

Main Methods:

• Leveraged GPT-4 to identify disease-related concepts and compute semantic similarity with neuroimaging patches.
• Reduced the influence of irrelevant patches based on semantic similarity.
• Constructed a graph of concepts and used a graph convolutional network (GCN) to extract structural information for prompting multimodal models.

Main Results:

• The proposed graph prompt learning method achieved superior performance in neurological disorder diagnosis compared to state-of-the-art approaches.
• The method effectively identified relevant image patches and utilized brain network structure for improved diagnostic accuracy.

Conclusions:

• The novel graph prompt learning approach offers a significant advancement in diagnosing neurological disorders using multimodal models.
• This method's ability to integrate semantic relevance and structural information holds promise for future clinical applications.