Comparative analysis of large language models on rare disease identification

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Rare Disease Diagnostics

Background:

• Diagnosing rare diseases presents significant challenges, including low prevalence, varied symptoms, and limited physician awareness, often resulting in diagnostic delays and misdiagnoses.
• The study investigates the diagnostic capabilities of advanced large language models (LLMs) for rare diseases, comparing their performance against human clinicians.

Discussion:

• Four LLMs (ChatGPT-4o, Claude 3.5 Sonnet, Gemini Advanced, Llama 3.1 405B) were evaluated on 152 real-world rare disease cases from the Chinese Medical Case Repository.
• LLMs demonstrated superior performance compared to human physicians in identifying rare diseases.
• Claude 3.5 Sonnet achieved the highest diagnostic accuracy (78.9%), significantly exceeding the human physician accuracy rate (26.3%).

Key Insights:

• LLMs can significantly enhance the accuracy and efficiency of rare disease diagnosis.
• Claude 3.5 Sonnet emerged as a leading AI tool for rare disease identification.
• AI-powered diagnostic support holds potential for improving healthcare, especially in resource-limited settings.

Outlook:

• Further clinical validation and rigorous testing are essential to confirm the efficacy and safety of LLMs in rare disease diagnostics.
• Addressing ethical considerations and data privacy is crucial for the responsible integration of AI tools into clinical practice.
• LLMs could serve as valuable assistive tools for clinicians, potentially reducing diagnostic odysseys for patients with rare conditions.