Multimodal Visual Question Answering System for Survival Prognosis in PSMA-Targeted Radiopharmaceutical Therapy

Large language models show great promise for clinical decision support but generally lack medical imaging integration. Therefore, vision-language models have been proposed, but there is little evidence on their performance, particularly compared with conventional machine learning. Prostate-specific membrane antigen (PSMA)-targeted radiopharmaceutical therapy is a well-established treatment for metastatic castration-resistant prostate cancer (mCRPC), but it is still not known which patients will be good long-term responders. Therefore, this study proposes a physician-inspired visual question answering (VQA) framework that emulates the structured integration of PSMA-directed imaging and clinical data used in routine clinical assessment for overall survival prognostication. Methods: As a proof of concept, we conducted a retrospective, multicenter study across 2 academic institutions that included 110 patients with mCRPC who were treated with either [¹⁷⁷Lu]Lu-PSMA I&T or [¹⁷⁷Lu]Lu-PSMA-617. Our VQA approach incorporates a 3-dimensional image encoder, the domain-specific language model RadBERT, and a cross-modal transformer to process baseline and interim (after 2 cycles) PET and SPECT images, clinical data, and clinical query. The model was trained to predict 24-mo survival, and performance was compared with state-of-the-art approaches. External validation was performed with the [¹⁷⁷Lu]Lu-PSMA-617 cohort. Results: The VQA model outperformed all state-of-the-art approaches while showing accuracy comparable to the query-free attention multimodal fusion model. Specifically, it achieved 81.8% accuracy (95% CI, 0.52-0.95) when integrating SPECT or PET images with prostate-specific antigen measurements. External validation of the VQA showed 71.9% accuracy (95% CI, 0.53-0.86) after few-shot fine-tuning, with further improvement over all state-of-the-art approaches in survival stratification. The VQA approach showed the closest calibration with the ground-truth survival distribution, with no significant differences observed (log-rank test, P = 0.76). Receiver operating characteristic analysis showed the highest area under the curve (0.75) among the evaluated models. Together, these findings suggest that the VQA approach demonstrated adaptability across different PSMA ligands while maintaining slightly improved yet comparable overall survival prognostic performance. Conclusion: By adopting a physician-inspired assessment workflow, the VQA framework shows advantages in integrating imaging and laboratory data for clinical decision support compared with conventional machine learning. The preliminary results indicate that VQA-based approaches warrant future evaluation for survival prognostication in patients with mCRPC who are receiving PSMA radiopharmaceutical therapy.