Retrieval-augmented in-context learning for multimodal large language models in disease classification

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Machine Learning

Background:

• Multimodal large language models (MLLMs) show promise in disease classification.
• In-context learning (ICL) in MLLMs can be limited by the quality of provided demonstrations.
• Dynamic retrieval of informative demonstrations is needed to improve MLLM performance.

Purpose of the Study:

• To enhance in-context learning (ICL) for multimodal large language models (MLLMs) in disease classification.
• To develop a framework that dynamically retrieves informative demonstrations.
• To improve the accuracy and efficiency of MLLMs in medical applications.

Main Methods:

• Proposed a Retrieval-Augmented In-Context Learning (RAICL) framework integrating retrieval-augmented generation (RAG) and ICL.
• Utilized patch embeddings and diverse encoders (ResNet, BERT, BioBERT, ClinicalBERT) for demonstration retrieval.
• Constructed optimized conversational prompts for ICL and evaluated on TCGA and IU Chest X-ray datasets across multiple MLLMs.

Main Results:

• RAICL consistently outperformed non-retrieval baselines, improving accuracy on TCGA (0.7857 to 0.8726) and IU Chest X-ray (0.7924 to 0.8658).
• Multimodal inputs were superior to single modalities, with text outperforming image; few-shot retrieval further enhanced performance.
• Euclidean distance as a similarity metric yielded the best results, with consistent improvements across various MLLMs, demonstrating robustness.

Conclusions:

• RAICL offers an efficient and scalable method to improve ICL in MLLMs for multimodal disease classification.
• The framework enhances MLLM capabilities in medical AI by dynamically selecting relevant data.
• RAICL represents a significant advancement in adapting MLLMs for complex clinical tasks.