Diagnosing Rare Genetic Diseases From Facial Images Using Visual Prompting With Multimodal Large Language Models: Diagnostic Accuracy Study

Rare genetic diseases often present with recognizable facial phenotypes. Whether multimodal large language models (MLLMs) can prioritize the diagnosis of rare genetic diseases from facial images and whether image enhancement or visual prompting can improve this process remain unclear.

This study aimed to evaluate the diagnostic accuracy of MLLMs for rare genetic diseases using facial images and to assess whether super-resolution enhancement or red-circle visual prompting improved performance.

This diagnostic accuracy study used 4746 facial images from the GestaltMatcher Database (GMDB), including 882 images in GMDB-Frequent from diseases with at least 6 images and 3864 images in GMDB-Rare from diseases with fewer than 6 images. Three locally deployed open-weight MLLMs-Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B-generated 50 ranked candidate diagnoses under raw image, CodeFormer-enhanced, Qwen-Image-Edit-2511-generated red-circle-prompted, mismatched red-circle, and 2-stage image-to-phenotype conditions. Candidate diagnoses were mapped to Mondo Disease Ontology (MONDO) identifiers. In addition, we evaluated a random baseline, visual-prompt quality, expert-annotated regions, repeated-inference stability, model-reported confidence, diagnostic-list perplexity, prediction overlap, and disease composition.

Top-50 accuracy ranged from 13.95% to 18.25% in GMDB-Frequent and from 0.57% to 1.35% in GMDB-Rare. Red-circle prompting produced numerical accuracy increases in 5 of 6 model-subset comparisons. Qwen3.8-27B showed a significant improvement in GMDB-Rare, with accuracy increasing from 0.57% to 1.27% (adjusted P=.009). In the good visual-prompt subgroup of GMDB-Frequent, Gemma-4-31B showed significant improvements over the super-resolution condition in both physician-rated analyses, with accuracy increases of 5.76 percentage points (Holm-adjusted P=.005) and 4.46 percentage points (Holm-adjusted P=.048), respectively. Matched red-circle inputs yielded higher accuracy than mismatched inputs across all model-subset comparisons, with adjusted P>.05. The image-to-phenotype workflow improved accuracy for Gemma-4-31B in GMDB-Frequent and Qwen3.8-27B in GMDB-Rare. In GMDB-Rare, the random baseline achieved 8.15% accuracy and exceeded all 3 models under the raw image and red-circle conditions. Prediction sets remained substantially overlapping after patient-image replacement, and top-1 predictions concentrated on a small number of diseases.

Under the evaluated experimental settings, the 3 MLLMs showed limited ability to diagnose rare genetic diseases from facial images, particularly diseases in GMDB-Rare. Visual prompting and image-to-phenotype diagnosis produced benefits in specific models and subgroups. The prediction-overlap, mismatched-image, and disease-composition analyses indicate that model outputs were only weakly constrained by patient-specific facial information. Further studies should evaluate physician comparisons, prospective multicenter data, and phenotype-grounded multimodal diagnostic workflows.