Assessing the Accuracy and Reliability of Large Language Models in Psychiatry Using Standardized Multiple-Choice...

Area of Science:

• Artificial Intelligence
• Psychiatry
• Medical Informatics

Background:

• Large language models (LLMs) show promise in mental health applications, from documentation to therapy.
• Assessing the accuracy and reliability of LLM psychiatric knowledge is vital for safe integration.
• Developing confidence measures for LLM responses is crucial for clinical use.

Purpose of the Study:

• Evaluate the accuracy and reliability of GPT-3.5, GPT-4, and GPT-4o using standardized psychiatry multiple-choice questions (MCQs).
• Identify predictors of accuracy for these large language models in a medical context.

Main Methods:

• A cross-sectional study tested three commercial LLMs (GPT-3.5, GPT-4, GPT-4o) on 150 psychiatry MCQs.
• Each model answered each question 10 times to assess accuracy and reliability.
• Analyzed accuracy, response consistency, and correlation with self-reported confidence.

Main Results:

• GPT-4 (84.0%) and GPT-4o (87.3%) significantly outperformed GPT-3.5 (58.0%) in accuracy.
• GPT-4 and GPT-4o showed comparable performance, while GPT-3.5 had lower response consistency.
• Response consistency correlated positively with accuracy; self-reported confidence did not reliably predict accuracy.

Conclusions:

• GPT-4 and GPT-4o possess accurate and reliable general psychiatric knowledge.
• Repeated prompting may serve as a method to gauge LLM response confidence.
• Findings support LLM potential in mental health, warranting further research in clinical settings.