在精神病学中使用标准化多选题评估大型语言模型的准确性和可靠性:横截面研究
Kaitlin Hanss1, Karthik V Sarma1, Anne L Glowinski1
1Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, United States.
Journal of medical Internet research
|May 20, 2025
概括
GPT-4和GPT-4o表现出准确可靠的精神病学知识,在标准化测试中表现优于GPT-3.5. 重复提示可能会增强对心理健康应用程序的大型语言模型响应的信心.
科学领域:
- 人工智能的人工智能
- 精神病学是一个精神病学.
- 医疗信息学 医疗信息学
背景情况:
- 大型语言模型 (LLM) 在心理健康应用中表现有前途,从文档到治疗.
- 评估LLM精神病学知识的准确性和可靠性对于安全的整合至关重要.
- 开发对LLM响应的信心指标对于临床使用至关重要.
研究的目的:
- 评估GPT-3.5,GPT-4和GPT-4o的准确性和可靠性,使用标准化精神病学多选择题 (MCQ).
- 在医疗环境中确定这些大型语言模型的准确性预测因素.
主要方法:
- 一项横截面研究对150个精神病学MCQ进行了测试,测试了三个商业LLM (GPT-3.5,GPT-4,GPT-4o).
- 每个模型都回答了每一个问题10次,以评估准确性和可靠性.
- 分析了准确性,响应的一致性和与自我报告的信心的相关性.
主要成果:
- 在精度方面,GPT-4 (84.0%) 和GPT-4o (87.3%) 显著超过了GPT-3.5 (58.0%).
- GPT-4和GPT-4o表现相似,而GPT-3.5的响应一致性较低.
- 响应的一致性与准确性有正相关;自我报告的信心无法可靠地预测准确性.
结论:
- GPT-4和GPT-4o拥有准确可靠的一般精神病学知识.
- 重复提示可以作为衡量LLM响应信心的一种方法.
- 这些发现支持了心理健康领域的LLM潜力,这证明了在临床环境中需要进一步的研究.
更多相关视频
相关概念视频
Self-Report Tests of Personality
282
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
282
Diagnostic and Statistical Manual of Mental Disorders (DSM)
35
The Diagnostic and Statistical Manual of Mental Disorders (DSM) serves as the primary classification system for mental health disorders, providing standardized diagnostic criteria for clinicians and researchers. First published by the American Psychiatric Association (APA) in 1952, the DSM has undergone several revisions to reflect evolving psychiatric understanding. The fifth edition, DSM-5, released in 2013, introduced key updates that expanded diagnostic categories and modified diagnostic...
35
Language and Cognition
302
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
302
Modeling in Therapy
39
Modeling, a key technique in therapy, uses observational learning to help clients acquire and practice new skills by watching therapists demonstrate desired behaviors. This approach, rooted in Albert Bandura's concept of vicarious learning, plays a significant role in therapeutic interventions for various psychological conditions, including social anxiety, ADHD, and depression.
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...
39


