Integrating Expert Knowledge into Large Language Models Improves Performance for Psychiatric Reasoning and Diagnosis

The authors sought to evaluate the performance of common large language models (LLMs) in psychiatric diagnosis, and the impact of integrating expert-derived reasoning on their performance. Clinical case vignettes and associated diagnoses were retrieved from the DSM-5-TR Clinical Cases book. Diagnostic decision trees were retrieved from the DSM-5-TR Handbook of Differential Diagnosis and refined for LLM use. Three LLMs were prompted to provide diagnosis candidates for the vignettes either by directly prompting or using the decision trees. These candidates and diagnostic categories were compared against the correct diagnoses. The positive predictive value (PPV), sensitivity, and F ₁ statistic were used to measure performance.

When directly prompted to predict diagnoses, the best LLM by F ₁ statistic (gpt-4o) had sensitivity of 77.6% and PPV of 43.3%. When making use of the refined decision trees, PPV was significantly increased (65.3%) without a significant reduction in sensitivity (71.8%). Across all experiments, the use of the decision trees statistically significantly increased the PPV, significantly increased the F ₁ statistic in 5/6 experiments, and significantly reduced sensitivity only for the category-based evaluation in 2/3 experiments.

When used to predict psychiatric diagnoses from case vignettes, direct prompting of the LLMs yielded most true positive diagnoses but had significant overdiagnosis. Integrating expert-derived reasoning into the process using decision trees improved LLM performance, primarily by suppressing overdiagnosis with minimal negative impact on sensitivity. This suggests that the integration of clinical expert-derived reasoning could improve the performance of LLM-based tools in the behavioral health setting.