Sequential sensitivity analysis of multimodal large language models for rare orbital disease detection

Delayed diagnosis of rare orbital diseases is attributed to limited clinical awareness. Building on prior evidence of multimodal large language models (MLLMs) for detecting common ocular conditions, this study aims to evaluate whether integrating multimodal clinical data can enhance the diagnostic accuracy of MLLMs for rare orbital diseases.

We conducted a multinational, multiracial, retrospective study. Two datasets were analyzed: Dataset 1, containing 6,786 single-eye photographs from China, was used to fine-tune a contrastive language-image pre-training (CLIP) for preliminary classification of healthy eyes, orbital diseases, and non-orbital diseases, and to compare its performance against three traditional models and three next-generation models. Dataset 2, comprising 170 participants from China, Singapore, and Thailand, was used to evaluate a MLLM (GPT-4o-Latest). Sequential sensitivity analysis assessed the impact of adding external eye photographs, chief complaints, racial information, and diagnostic reasoning prompts. An AI agent combining the CLIP model with GPT-4o-Latest was further evaluated. The model's ability to generate medical reports and examination recommendations was also assessed.

Here we show that the CLIP model achieves 90.21% preliminary detection accuracy, surpassing all baseline models. MLLM detection accuracy improves significantly with the inclusion of multimodal inputs. When relying on external eye images, the top-5 accuracy is 25.68%. The combined agent raises top-5 accuracy to 85.29%. Generated reports and recommendations display high accuracy, readability, completeness, and low potential for harm.

Our study demonstrates the potential of MLLM in improving diagnostic accuracy and supporting clinical decision-making for rare orbital diseases.