Diagnostic Accuracy of Large Language Models for Rare Diseases: A Systematic Review and Meta-Analysis

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Rare Disease Diagnostics

Background:

• Evidence on large language models (LLMs) for rare disease diagnosis is fragmented.
• LLMs show potential but require rigorous evaluation for clinical use.
• Systematic review and meta-analysis to assess LLM diagnostic accuracy for rare diseases.

Purpose of the Study:

• Synthesize evidence on LLM diagnostic performance in rare diseases.
• Identify factors influencing LLM accuracy (e.g., knowledge augmentation, input modality).
• Evaluate the readiness of LLM technology for clinical translation.

Main Methods:

• Systematic search of major databases (PubMed, Embase, etc.) and preprint servers.
• Inclusion of studies evaluating LLM-based rare disease differential diagnosis.
• Meta-analysis of Recall@1 (R@1) using random-effects models; subgroup and post-hoc analyses.

Main Results:

• Pooled R@1 was 43.3% across 19 system-dataset entries (N=39,529 cases).
• Augmented LLMs (52.5% R@1) outperformed standalone LLMs (35.4% R@1).
• Performance varied by benchmark composition; higher R@1 with fewer ultra-rare diseases.

Conclusions:

• LLM diagnostic performance for rare diseases is inconsistent and influenced by benchmark characteristics.
• Augmented LLMs and simpler disease profiles improve accuracy.
• High risk of bias and lack of prospective validation necessitate caution before clinical deployment.