自动化观察者 OPTION-5 共同决策的测量:通过将大型语言模型与人类评级进行比较来评估有效性
Sai P Selvaraj1, Renata W Yen2, Rachel Forcino3
1Anterior, Inc., New York, USA.
Patient education and counseling
|September 28, 2025
概括
大型语言模型 (LLM) 在评估临床咨询中共享决策方面表现有前途. 这些人工智能工具可以评估临床医生与患者的对话,从而有可能改善医疗保健专业人员的反.
科学领域:
- 医疗保健中的人工智能
- 临床沟通分析 临床沟通分析
- 医学决策研究 医学决策研究
背景情况:
- 基于观察者的共享决策措施是资源密集的,因为依赖于人类评估者,限制了例行评估.
- 生成型人工智能 (AI) 提供了提高评估速度和准确性的潜力,同时降低了人力负担.
研究的目的:
- 评估来自Gemini,GPT和LLaMA家族的大型语言模型 (LLM) 的性能.
- 评估LLM在乳腺癌早期手术咨询中评估共享决策的能力.
主要方法:
- 将LLM生成的分数与使用观察者OPTION-5衡量的训练有素的人类评分器进行了比较.
- 分析了287份匿名的乳腺癌咨询记录.
- 测试了各种提示,并评估了LLMs区分高与低得分遭遇的能力.
主要成果:
- GPT-4o和Gemini-1.5-Pro-002的分数与人类评分相关 (皮尔森r ≈ 0.6,p < 0.01).
- 士学位的表现大约是人类评审者之间达成的75-80%的协议 (r = 0.77).
- 在此过程中,LLM成功地区分了高分和低分的比赛 (t > 10,p < 0.01).
结论:
- 法律学士可以使用现有的共享决策措施来评估临床医生与患者之间的对话.
- 快速开发和微调对于优化LLM性能至关重要.
- 未来的研究应该专注于概括性和更大的数据集,以提高模型性能.
相关概念视频
01:14Decision Making: Traditional Method
5.1K
The process of hypothesis testing based on the traditional method includes calculating the critical value, testing the value of the test statistic using the sample data, and interpreting these values.
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
5.1K
01:11Observational Studies
10.7K
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
10.7K