Trustworthiness Evaluation of Medical Vision-Language Models: A Scoping Review and a Proposed Minimum Evaluation...

Medical vision-language models combine images and text to answer questions, generate reports, and support image interpretation. Task accuracy alone does not establish whether their outputs depend on relevant visual evidence or remain reliable under changes in the input.

We aimed to describe how trustworthiness is evaluated in medical vision-language models, identify gaps in evaluation and reporting, and develop a preliminary minimum evaluation checklist.

We conducted a scoping review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews. We searched PubMed/MEDLINE, Embase, Scopus, Web of Science Core Collection, and IEEE Xplore for English-language publications from January 1, 2022, to March 25, 2026. Eligible reports were peer-reviewed full papers that evaluated an image-and-language model in a medical imaging context and reported a measured result on at least one of eight trustworthiness dimensions. One reviewer charted the studies and a second verified the extraction. We summarized evaluation practices descriptively and applied the proposed eight-item Minimum Trustworthiness Evaluation Checklist (MiTEC) retrospectively. We did not perform a risk-of-bias appraisal.

: We identified 516 records, screened 506 database records at title and abstract, and assessed 72 unique reports after merging duplicate full-text entries. The revised synthesis includes 29 studies: 11 journal articles and 18 conference papers. Robustness was evaluated in 11/29 (37.9%), hallucination in 7/29 (24.1%), distribution shift in 6/29 (20.7%), and visual grounding in 5/29 (17.2%). Interpretability and fairness were each evaluated in 4/29 (13.8%); uncertainty or confidence reliability in 2/29 (6.9%); and safety in 1/29 (3.4%). Two studies used a text-only baseline, and four used at least one image-reliance control. No included study was confirmed to report a formal calibration metric or a selective-prediction curve. Retrospective MiTEC scoring yielded 41 reported items among 232 study-item pairs (17.7%).

Evaluation practices varied across the included studies. Image-reliance controls, formal calibration measures, and subgroup analyses were uncommon. MiTEC is an author-proposed item set that requires independent testing and stakeholder consensus. The findings describe the included sample; incomplete linkage of preprints to published versions limits claims about the wider literature.