Retrieval-augmented in-context learning for multimodal large language models in disease classification

We aim to dynamically retrieve informative demonstrations, enhancing in-context learning in multimodal large language models (MLLMs) for disease classification.

We propose a Retrieval-Augmented In-Context Learning (RAICL) framework, which integrates retrieval-augmented generation (RAG) and in-context learning (ICL) to adaptively select demonstrations with similar disease patterns, enabling more effective ICL in MLLMs. Specifically, RAICL examines embeddings from patch embeddings and diverse encoders, including ResNet, BERT, BioBERT, and ClinicalBERT, to retrieve appropriate demonstrations, and construct conversational prompts optimized for ICL. We evaluated the framework on two real-world multi-modal datasets (TCGA and IU Chest X-ray), assessing its performance across multiple MLLMs (Qwen, Llava, Gemma), embedding strategies, similarity metrics, and varying numbers of demonstrations.

RAICL consistently outperformed non-retrieval baselines. Accuracy improved from 0.7857 to 0.8726 on TCGA and from 0.7924 to 0.8658 on IU Chest X-ray. Multimodal inputs surpassed single modalities, with text-only outperforming image-only, and few-shot retrieval further boosted performance. Across similarity metrics, Euclidean distance achieved the best results. Improvements were consistent across different MLLMs, demonstrating robustness and scalability.

RAICL provides an efficient and scalable approach to enhance in-context learning in MLLMs for multimodal disease classification.