Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis

Area of Science:

• Artificial Intelligence in Medicine
• Computational Psychiatry
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) show potential in assisting with psychiatric diagnoses.
• Evaluating LLM performance in clinical settings requires rigorous assessment.
• The impact of structured clinical reasoning on LLM diagnostic accuracy is not well-defined.

Purpose of the Study:

• To assess the diagnostic performance of common LLMs using clinical case vignettes.
• To investigate the effect of integrating expert-derived diagnostic decision trees on LLM performance.
• To compare direct LLM prompting versus LLM-assisted reasoning for psychiatric diagnosis.

Main Methods:

• Retrieved clinical case vignettes and diagnoses from DSM-5-TR resources.
• Developed and refined diagnostic decision trees for LLM implementation.
• Prompted three LLMs with vignettes, using both direct and decision-tree-guided approaches.
• Evaluated performance using positive predictive value (PPV), sensitivity, and F1 statistic.

Main Results:

• Direct LLM prompting resulted in high sensitivity but low positive predictive value (PPV), indicating overdiagnosis.
• Utilizing decision trees significantly increased PPV (from 40.4% to 65.3%) with a modest decrease in sensitivity (from 76.7% to 70.9% for the best model).
• Decision trees statistically improved PPV and F1 scores in most experiments, while slightly reducing sensitivity.

Conclusions:

• Direct LLM prompting for psychiatric diagnosis leads to overdiagnosis.
• Integrating expert-derived decision trees enhances LLM diagnostic performance by reducing overdiagnosis.
• LLM-assisted reasoning shows promise for improving clinical decision support tools in behavioral health.