PsychiatryBench: a multi-task benchmark for LLMs in psychiatry.

Area of Science:

• Artificial Intelligence in Medicine
• Psychiatric Diagnostics and Treatment

Background:

• Large language models (LLMs) show promise for psychiatric practice, aiding diagnosis, documentation, and therapy.
• Current LLM evaluations use limited data, lacking clinical validity and failing to assess complex diagnostic reasoning.

Purpose of the Study:

• Introduce PsychiatryBench, a novel benchmark for evaluating LLMs in psychiatry.
• Ensure rigorous evaluation using expert-validated psychiatric textbooks and casebooks.

Main Methods:

• Developed PsychiatryBench with 5,188 expert-annotated items across eleven psychiatric QA tasks.
• Evaluated frontier LLMs (e.g., Gemini, GPT-5) and medical models (e.g., MedGemma) using standard metrics and LLM-as-judge.
• Tasks included diagnostic reasoning, treatment planning, and longitudinal follow-up.

Main Results:

• Identified significant gaps in LLM clinical consistency and safety, especially in multi-turn follow-up and management.
• Frontier LLMs and medical models showed limitations in complex psychiatric reasoning.
• PsychiatryBench provides a comprehensive assessment of LLM capabilities in mental health.

Conclusions:

• Current LLMs require specialized tuning and improved evaluation for safe and effective psychiatric applications.
• PsychiatryBench serves as a critical tool for advancing AI in mental healthcare.
• The benchmark highlights the need for robust evaluation paradigms beyond simple Q&A.