Integrating Expert Knowledge into Large Language Models Improves Performance for Psychiatric Reasoning and Diagnosis

Area of Science:

• Artificial Intelligence in Medicine
• Computational Psychiatry
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) are increasingly explored for clinical applications.
• Accurate psychiatric diagnosis is complex and benefits from structured reasoning.
• Evaluating LLM performance in diagnostic tasks is crucial for potential integration.

Purpose of the Study:

• To assess the diagnostic performance of common LLMs using clinical case vignettes.
• To investigate the impact of integrating expert-derived diagnostic decision trees on LLM performance.
• To compare direct LLM prompting versus decision tree-assisted diagnosis.

Main Methods:

• Retrieved clinical case vignettes and diagnoses from DSM-5-TR resources.
• Developed and refined diagnostic decision trees for LLM implementation.
• Prompted three LLMs with and without decision trees to generate psychiatric diagnoses.
• Evaluated performance using positive predictive value (PPV), sensitivity, and F ₁ statistic.

Main Results:

• Direct LLM prompting (gpt-4o) achieved 77.6% sensitivity and 43.3% PPV.
• Decision tree integration significantly increased PPV to 65.3% with maintained sensitivity (71.8%).
• Decision trees improved F ₁ statistic in most experiments and reduced overdiagnosis.

Conclusions:

• Integrating expert-derived reasoning via decision trees enhances LLM performance in psychiatric diagnosis.
• This approach primarily mitigates overdiagnosis, a key limitation of direct LLM prompting.
• LLM-based tools augmented with clinical reasoning show potential for improving behavioral health diagnostics.