聊天GPT的准确性和可重复性基于一组客观听力测试的多选题
Krzysztof Kochanek1, Henryk Skarzynski2, Wiktor W Jedrzejczak1
1Department of Experimental Audiology, Institute of Physiology and Pathology of Hearing, Warsaw, POL.
Cureus
|June 10, 2024
概括
与ChatGPT 3.5.5相比,ChatGPT 4在客观听力测试中显示出更高的准确性和可重复性. 然而,响应的变化引起了人们对在听力学中专业使用的担忧.
科学领域:
- 听力学和听力科学 听力学和听力科学
- 医疗保健中的人工智能
- 目标听力测量测量方法
背景情况:
- 有限的研究存在于人工智能 (AI) 聊天机器人,如ChatGPT在听力学和听力诊断中.
- 在这个领域,人工智能聊天机器人响应的一致性和可重复性尚未得到充分证实.
研究的目的:
- 评估ChatGPT 3.5和ChatGPT 4在回答客观听力测试问题时的准确性.
- 在一周的时间内评估两个ChatGPT版本的响应的短期可重复性.
主要方法:
- 使用了30个客观听力测试课程的多选择题.
- 在四个不同的日子里,向ChatGPT 3.5和ChatGPT 4提出了五次问题.
- 响应的准确性与关键相比较;可重复性用百分比协议和科恩的卡帕来衡量.
主要成果:
- 聊天GPT 4实现了65-69%的准确性,显著超过了聊天GPT 3.5的48-49%的准确性.
- 在一天内重复性对于ChatGPT 4 (87-88%的同意) 比ChatGPT 3.5 (76-79%的同意) 高.
- 日间的重复性也有利于ChatGPT 4 (85-88%的同意) 与ChatGPT 3.5 (75-79%的同意).
结论:
- 与ChatGPT 3.5相比,ChatGPT 4在客观听力测试问题上显示出更高的准确性和可重复性.
- 尽管有所改进,但这两种人工智能模型的显著响应变化质疑它们在听力学中的直接专业适用性.
相关概念视频
01:17Cochran's Q Test
298
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
298
01:13Accuracy and Errors in Hypothesis Testing
196
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
196