Simulated Reasoning and Self-Verification in Generalist Large Language Models for Psychiatric Diagnostic Performance:...

Large language models (LLMs) have rapidly garnered significant interest for application in psychiatry and behavioral health. However, recent studies have identified significant shortcomings and potential risks in the performance of LLM-based systems, complicating their application to psychiatric diagnosis. Two promising approaches to addressing these challenges and improving the efficacy of these models are simulated reasoning and self-verification, in which additional "reasoning tokens" are used to guide model output, either during or after inference.

We aimed to explore how the use of simulated reasoning (via the use of large reasoning models, or LRMs) and self-verification (via supplemental prompting) affect the psychiatric diagnostic performance of language models.

106 case vignettes and associated diagnoses were extracted from the DSM-5-TR Clinical Cases book, with permission. An LLM and an LRM model were selected from each of the two vendors studied (OpenAI and Google). Two inference approaches were developed, a Basic approach that directly prompted models to provide diagnoses, and a Self-Verification approach that augmented the Basic approach with additional prompts. All case vignettes were processed by the four models and two approaches, and diagnostic performance was evaluated using the sensitivity and positive predictive value (PPV). Linear mixed effect models were used to test for significant differences between the vendors, model types, and inference approaches.

All vignettes were successfully processed by each model and inference approach. Sensitivity ranged from 0.732 to 0.817, and PPV ranged from 0.534 to 0.779. The best overall performance was found in o3-pro using SV, with a sensitivity of 0.782 and a PPV of 0.779. No statistically significant fixed effects were found for sensitivity. For PPV, a statistically significant effect was found for prompt type (SV - coefficient 0.09, p=0.002), model type (LRM - coefficient 0.09, p=0.003), and the interaction between model type and vendor (LRM:OpenAI - coefficient 0.10, p=0.021).

We found that both simulated reasoning and self-verification yielded statistically significant improvements in the PPV, without significant differences in the sensitivity. The addition of the manually specified self-verification prompt improved the PPV even when simulated reasoning was used. This suggests that future efforts to apply language models in behavioral health may benefit from a combination of manually crafted reasoning prompts and automated simulated reasoning.