PsyEval: a comprehensive large language model evaluation benchmark for mental health.

Area of Science:

• Artificial Intelligence
• Mental Health Technology
• Computational Psychology

Background:

• Evaluating large language models (LLMs) in mental health is challenging due to symptom subjectivity and context dependency.
• Existing benchmarks may not adequately capture the nuances of mental health assessment and support.

Purpose of the Study:

• Introduce PsyEval, a novel benchmark for assessing LLMs in mental health.
• Evaluate the performance of eleven advanced LLMs using the PsyEval benchmark.
• Investigate the impact of prompting strategies on LLM responses in mental health contexts.

Main Methods:

• Developed PsyEval, a benchmark focusing on knowledge, diagnosis, and emotional support in mental health.
• Administered PsyEval to eleven diverse LLMs.
• Employed various prompting strategies to test model robustness and adaptability.

Main Results:

• Significant performance gaps were identified in LLMs' ability to reason accurately within mental health scenarios.
• LLMs demonstrated limitations in providing appropriate and sensitive responses.
• Prompting strategies showed a variable impact on model performance across different tasks.

Conclusions:

• Current LLMs require substantial enhancement for reliable application in mental health.
• PsyEval offers a structured framework for future LLM development and evaluation in this domain.
• Further research is needed to improve LLM's contextual understanding and empathetic capabilities for mental health.