[Diagnostic agreement of large language models with clinical reference diagnoses of mental disorders and associated...

Insufficient diagnostic evidence and blurred diagnostic boundaries may compromise the reliability of clinical judgment and research labels, whereas standardized diagnostic interviews are resource-intensive and difficult to implement widely. Routinely archived inpatient medical records may therefore provide a practical basis for assisted diagnostic review. This study aims to evaluate the agreement of several commonly used general-purpose large language models (LLMs) with clinical reference diagnoses and the stability of their outputs across repeated runs when reviewing psychiatric admission records, analyze the associations of diagnostic information completeness and diagnostic complexity with Top-1 diagnostic agreement, and explore the characteristics of cases with frequent Top-1 disagreement to inform subsequent manual review.

A retrospective diagnostic agreement study was conducted using 125 psychiatric inpatient records. The clinical reference diagnoses belonged to 3 Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR) diagnostic categories: schizophrenia spectrum disorders, bipolar disorders, and depressive disorders. The models received only standardized and de-identified admission-stage medical record text. GPT-5.5, Gemini 3.5 Flash, and Qwen 3.7 Max independently assessed diagnostic information completeness and diagnostic complexity for all cases in 3 separate runs. Ten candidate models were first preliminarily screened using a stratified sample of 45 cases. Six models, Qwen 3.7 Max, GPT-5.5, Gemini 3.5 Flash, Qwen 3.6 35B-A3B, DeepSeek-R1-Distill-Qwen-32B, and Gemma 4 26B-A4B, were subsequently selected for formal evaluation. Each model independently evaluated all 125 cases in 3 runs, yielding a total of 2 250 diagnostic judgments. The primary outcome was the Top-1 agreement rate; secondary outcomes included the Top-2 agreement rate and cross-run consistency. A multivariable linear probability model was used to analyze factors associated with Top-1 agreement, and cases with frequent Top-1 disagreement were subjected to exploratory retrospective review.

Across all 3 runs of the 6 models, the overall Top-1 agreement rate was 93.0% (2 092/2 250; 95% CI 89.0% to 96.4%), and the Top-2 agreement rate was 95.6% (2 151/2 250). Cross-run consistency exceeded 92.0% for all models. Based on 375 diagnostic judgments per model across the 3 runs, Qwen 3.7 Max had the numerically highest Top-1 agreement rate among online models, at 94.4%, whereas Qwen 3.6 35B-A3B had the numerically highest Top-1 agreement rate among locally deployed models, at 93.1%. In the multivariable analysis based on all 6 models and 3 runs, after adjustment for diagnostic information completeness, model, diagnostic category, and run, each 1 standard deviation increase in diagnostic complexity was associated with an average 3.11-percentage-point decrease in Top-1 agreement (95% CI -5.66 to -0.57; P=0.017). Diagnostic information completeness showed a positive trend toward association with Top-1 agreement, but the association did not reach statistical significance (P=0.075). Seven cases with frequent Top-1 disagreement were identified. Exploratory retrospective review suggested that the diagnostic disagreements may involve the temporal relationship between mood and psychotic symptoms, integration of cross-sectional presentation with longitudinal course, and missing or ambiguous key information required for differential diagnosis.

At the level of the 3 diagnostic categories of schizophrenia spectrum disorders, bipolar disorders, and depressive disorders, general-purpose LLMs showed high agreement with clinical reference diagnoses and relatively stable outputs across repeated runs. Greater diagnostic complexity was associated with lower Top-1 agreement. Cases with frequent Top-1 disagreement may serve as exploratory signals for manual review but should not be directly interpreted as indicating model diagnostic errors or clinical misdiagnosis. The practical value of such signals requires further validation through independent expert assessment and external samples.