从人工智能模型获得的答案是否可用于信息目的,可重复?
Yasemin Tunca1, Volkan Kaplan2, Murat Tunca1
1Department of Orthodontics, Faculty of Dentistry, Kutahya Health Sciences University, Kutahya, Turkey.
International orthodontics
|October 4, 2025
概括
大型语言模型显示,正牙答案的重复性各不相同. 虽然准确,但一些AI模型缺乏时间一致性,影响临床使用.
科学领域:
- 人工智能在牙科中的应用
- 自然语言处理自然语言处理.
- 临床决策支持 临床决策支持
背景情况:
- 评估人工智能生成的正牙信息的可靠性对于临床整合至关重要.
- 大型语言模型 (LLM) 越来越多地使用,需要评估它们的一致性.
研究的目的:
- 评估多个LLM随着时间的推移对正义牙反应的重复性.
- 为了比较ChatGPT-3.5,ChatGPT-4.0,Gemini和Gemini-Advanced的时间稳定性.
主要方法:
- 四位LLM (ChatGPT-3.5,ChatGPT-4.0,Gemini,Gemini-Advanced) 在三个时间点回答了40个正牙问题.
- 答案由两名正牙专家独立评估,使用3分准确度表.
- 计算了评价者间一致性 (科恩卡帕) 和模型重复性 (ICC);使用弗里德曼和斯皮尔曼测试分析了时间差异.
主要成果:
- 观察到实质性的评价者间一致性 (卡帕:0.624-0.749).
- 重复性差异很大,ICC值在0.666 (Gemini) 到0.960 (ChatGPT-3.5) 之间.
- 随着时间的推移,发现了模型准确度的显著差异 (P<0.001),时间点之间的正相关性较弱 (ρ=0.284-0.383).
结论:
- 在AI正牙反应的时间重复性方面存在统计学上显著的差异.
- 一些LLM表现出高精度,但随着时间的推移,一致性有限.
- 评估准确性和时间稳定性对于在牙矯正中临床AI实施至关重要.
相关概念视频
Measures of Intelligence
8.3K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
8.3K
Statistical Analysis: Overview
14.5K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
14.5K
Non-equilibrium in the Cell
5.3K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
5.3K
Accuracy, limits, and approximation
1.1K
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
1.1K
Accuracy and Precision
14.0K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. Highly accurate...
14.0K
Stereotype Content Model
15.3K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
15.3K
