Simulated Reasoning and Self-Verification in Generalist Large Language Models for Psychiatric Diagnostic Performance:...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Psychiatry
• Natural Language Processing

Background:

• Large language models (LLMs) show promise for psychiatry but have limitations in diagnostic accuracy.
• Potential risks and shortcomings complicate the application of LLM-based systems in psychiatric diagnosis.
• Simulated reasoning and self-verification are emerging techniques to enhance LLM performance.

Purpose of the Study:

• To evaluate the impact of simulated reasoning (using large reasoning models, LRMs) and self-verification on LLM psychiatric diagnostic performance.
• To compare diagnostic accuracy metrics (sensitivity and positive predictive value) between different models and inference approaches.

Main Methods:

• Extracted 106 DSM-5-TR case vignettes and diagnoses.
• Utilized LLMs and LRMs from two vendors (OpenAI, Google) with Basic and Self-Verification inference approaches.
• Evaluated diagnostic performance using sensitivity and positive predictive value (PPV), analyzed with linear mixed effect models.

Main Results:

• Sensitivity ranged from 0.732 to 0.817; PPV ranged from 0.534 to 0.779.
• Self-verification and LRM models significantly improved PPV (p<0.003).
• An interaction effect was found between model type and vendor for PPV (LRM:OpenAI, p=0.021).

Conclusions:

• Both simulated reasoning and self-verification significantly improved PPV for psychiatric diagnosis, with no significant change in sensitivity.
• The addition of self-verification prompts enhanced PPV, even when simulated reasoning was employed.
• Combining manually crafted prompts with automated simulated reasoning may optimize LLM application in behavioral health.