Development of a multimodal large language model for early warning and diagnosis of chronic ocular GVHD

Chronic ocular graft-versus-host disease (coGVHD) after allogeneic hematopoietic stem cell transplantation (allo-HSCT) may lead to irreversible ocular surface damage and even vision loss. Current management of coGVHD faces challenges, with frequent missed or misdiagnosed cases. This study aimed to leverage a multimodal large language model (MLLM) to develop an early warning and diagnostic system for coGVHD. A total of 666 post-allo-HSCT patients (early warning model) and 805 post-allo-HSCT patients (1574 eyes, diagnostic model) were enrolled for construction, internal validation, and external validation of the corresponding models. We proposed the GVHD-MLLM, a multitask multimodal network that fused latent representations from four modal sequences to provide high-precision, real-time predictions for two tasks. The GVHD-MLLM achieved high performance in internal testing, with AUROCs of 93.44% (95% CI: 91.85-95.03%) for early warning, 98.98% (95% CI: 98.59-99.36%) for diagnosis, and 98.24% (95% CI: 98.05-98.43%) for disease severity grading. In external validation, the early warning AUROC was 83.45%, while diagnostic AUROCs across three external sites were all above 96.0%. The disease severity of patients seeking medical treatment after using the early warning model was significantly lower. Junior ophthalmologists also improved diagnostic accuracy using the model as an auxiliary tool. The GVHD-MLLM can process rich multi-modal information collected in clinical practice, and is expected to become an effective tool for managing coGVHD.