开发和验证基于GPT的评估器,使用Gap-Kalamazoo沟通技能评估表
Yu-Jeng Ju1, Yi-Ching Wang1, Shih-Chieh Lee1,2
1School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan.
Medical teacher
|August 23, 2025
概括
一个生成预训练变压器 (GPT) 评分器被开发来评估沟通技巧. 虽然它在低风险环境中显示出作为总分的补充工具的潜力,但其域级性能有限.
科学领域:
- 自然语言处理
- 医疗保健中的人工智能
- 沟通技能的评估
背景情况:
- 在医疗保健中,准确评估沟通技巧至关重要.
- 传统的评估方法可能耗时且主观.
- 自动化工具为高效和一致的评估提供了潜力.
研究的目的:
- 开发和评估基于预先训练的转换器 (GPT) 的评分器来评估沟通技巧.
- 与人类评估相比,检查GPT评价者的互评价可靠性和并发有效性.
- 通过Gap-Kalamazoo沟通技巧评估表格 (GKCSAF) 确定GPT评级器在评估治疗师与患者之间的互动中的有用性.
主要方法:
- 使用GKCSAF框架开发了一个基于GPT的评级器.
- 评估了80个先前由人类评估者评估的治疗师-患者交互记录.
- 使用平均绝对误差 (MAE) 和类内相关系数 (ICC) 分析了评级者之间的可靠性.
- 通过将GPT得分与人类平均得分进行比较来评估并发有效性.
主要成果:
- 对于总分数 (12. 2% - 21. 0%) 的MAE%是可以接受的,但在人体评分中ICC为0. 00- 0. 35).
- 域级分析显示只有四个领域的MAE%是可接受的,所有九个领域的一致性很差 (加权 κ ≤ 0.38).
- 同时有效性也显示了可接受的MAE%,但在总和域级的ICC值较低.
结论:
- 在低风险的沟通技巧评估中,GPT评分器是一个有前途的补充工具.
- 目前域级的业绩有限,需要谨慎进行详细或高风险的评估.
- 需要进一步改进,以提高GPT评分器的准确性和可靠性,以进行全面的沟通技能分析.
更多相关视频
09:16Use of a Video Scoring Anchor for Rapid Serial Assessment of Social Communication in Toddlers
Published on: March 14, 2018
10.3K
09:00Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
Published on: August 16, 2024
909
相关概念视频
01:03Data Validation
5.3K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
5.3K
01:22Self-Report Tests of Personality
454
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
454
01:29Reliability and Validity
13.2K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.2K
01:21Barriers to Effective Communication II
3.8K
The barriers to effective communication also include cultural barriers, semantic barriers, gender barriers, and time constraints.
Cultural barriers:
Differences in values, beliefs, religion, knowledge, and tradition can significantly impact communication. Awareness of nonverbal cues is critical, especially when conversing with a patient from a different culture. What appears appropriate in one culture may be inappropriate in another.
Semantic barriers:
As a result of their tendency to use...
Cultural barriers:
Differences in values, beliefs, religion, knowledge, and tradition can significantly impact communication. Awareness of nonverbal cues is critical, especially when conversing with a patient from a different culture. What appears appropriate in one culture may be inappropriate in another.
Semantic barriers:
As a result of their tendency to use...
3.8K