准确性是不够的:在整形大语言模型 (LLM) 应用中推理和参考可靠性
Shashwat Singh1, Pranav Chandrasekhar2
1Trauma and Orthopaedics, The Queen Elizabeth Hospital King's Lynn NHS Foundation Trust, King's Lynn, GBR.
Cureus
|February 9, 2026
概括
像GPT-5这样的大型语言模型在骨科检查中显示出高精度,但往往是伪造引用. 评估推理和证据至关重要,而不仅仅是正确的答案.
科学领域:
- 人工智能在医学中的应用
- 医疗教育 技术 技术 医学教育
- 整形外科手术培训班 整形外科手术培训班
背景情况:
- 大型语言模型 (LLM) 展示了与骨科研究生实习生相匹配的性能.
- 临床医生越来越多地使用LLM用于教育和决策支持目的.
- 目前的LLM评估重点是准确性,忽视了推理质量和参考可靠性.
研究的目的:
- 在研究生骨科检查中系统地评估LLM的答案准确性,推理质量和参考可靠性之间的关系.
- 评估GPT-5在2024年整形外科培训内检查 (OITE) 的表现.
主要方法:
- 在 2024 年的 OITE (203 个问题) 中,GPT-5 提供了答案,理由和参考资料.
- 精度与美国骨科外科医生学会 (AAOS) 数据进行了比较.
- 一个子样本经历了推理和引用的详细验证,将GPT-5的推理与AAOS解释进行了比较.
主要成果:
- GPT-5的准确率达到了78.3%,超过了OITE通过门和PGY-5居民分数.
- 幻觉发生在33%的答案中,错误答案 (50%) 与正确答案 (15.9%) 相比显著高.
- 对正确答案的推理很高,95.5%与AAOS解释相匹配;然而,所有答案中有33%引用了伪造或误导的引用.
结论:
- 在OITE上,GPT-5的准确性很高,但其参考可靠性较差.
- 即使是准确的答案也可能取决于有缺陷或无法验证的来源.
- 在医学教育中评估LLM需要评估推理和证据验证以及准确性.
相关概念视频
Reason and Intuition
7.5K
The human brain processes information for decision-making using one of two routes: an intuitive system and a rational system (Epstein, 1994; popularized by Kahneman, 2011 as System 1 and System 2, respectively). The intuitive system is quick, impulsive, and operates with minimal effort, relying on emotions or habits to provide cues for what to do next, while the rational system is logical, analytical, deliberate, and methodical. Research in neuropsychology suggests that the...
7.5K
Reliability and Validity
14.1K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.1K
Reasoning
440
Reasoning is the action of thinking about something in a logical, sensible way. It is integral to problem-solving, decision-making, and critical thinking. Reasoning can be inductive or deductive. Reasoning involves transforming information into conclusions, which is essential for problem-solving, decision-making, and critical thinking.
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
440
Deductive Reasoning
69.1K
Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning, which means that it uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid.
For example, a researcher can deduce specific predictions...
For example, a researcher can deduce specific predictions...
69.1K
Language
919
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
919
Inductive Reasoning
67.9K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
67.9K


