Benchmarking large language models against practicing clinicians on psychopathological assessment.

Area of Science:

• Psychiatry
• Artificial Intelligence
• Computational Linguistics

Background:

• Psychiatric assessment heavily relies on language, making Large Language Models (LLMs) a potential tool.
• Structured, item-level assessments from clinical interviews using LLMs are under-researched.

Purpose of the Study:

• To evaluate the accuracy of LLMs in psychopathological assessment using structured interview data.
• To compare LLM performance against early-career clinicians and explore their error profiles and utility in disagreement resolution.

Main Methods:

• Ten LLMs assessed transcripts of three simulated psychiatric interviews against the 100-item Association for Methodology and Documentation in Psychiatry (AMDP) system.
• LLM performance was benchmarked against 108 early-career clinicians using audiovisual recordings, with expert consensus as the reference.
• Error profiles and simulated disagreement resolutions between LLMs and clinicians were analyzed.

Main Results:

• GPT-5.1 and Gemini-3-Pro-Preview achieved the highest accuracy (0.72), comparable to the 64th percentile of clinicians.
• GPT-5.1 demonstrated strong performance across depression (0.81), mania (0.76), and schizophrenia (0.60) scenarios.
• LLMs flagged more items as 'not assessable' (19.4%) compared to clinicians (11.4%), especially for observation-dependent items.

Conclusions:

• LLMs show potential as a tool for structured psychopathological assessment, achieving accuracy comparable to clinicians in simulated settings.
• LLMs exhibit different error patterns than clinicians, potentially offering complementary insights.
• LLM assistance in disagreement resolution improved accuracy, suggesting their utility in clinical decision support, though validation in real-world settings is crucial.