五个先进的聊天机器人解决欧洲放射学文凭 (EDiR) 基于文本的问题:性能和一致性的差异
Jakub Pristoupil1, Laura Oleaga2, Vanesa Junquero3
1Department of Imaging Methods, Motol University Hospital and Second Faculty of Medicine, Charles University, Prague, Czech Republic.
European radiology experimental
|August 19, 2025
概括
克劳德 3.5 索内特在欧洲放射学文凭 (EDiR) 问题的表现,信心和一致性方面领先. 所有测试的AI聊天机器人在这些EDiR考试问题上都超过了人类的表现.
科学领域:
- 医疗教育中的人工智能
- 放射学中的大型语言模型
- 聊天机器人性能评估评估
背景情况:
- 在医学检查中评估人工智能聊天机器人的能力.
- 评估聊天机器人的性能,信心和响应的一致性.
研究的目的:
- 为了比较五种大型语言模型聊天机器人在解决欧洲放射学文凭 (EDiR) 多重答案问题的表现.
- 为了确定哪个聊天机器人表现出卓越的准确性,信心和响应一致性.
主要方法:
- 五个聊天机器人 (ChatGPT-4o,ChatGPT-4o-mini,Copilot,Gemini,Claude 3.5 Sonnet) 在52个基于文本的EDiR多重响应问题上进行了测试.
- 聊天机器人提供了答案,信任得分 (0-10),并在两个代中进行了评估.
- 一个加权公式计算了每个问题的分数 (0.0-1.0).
主要成果:
- 克劳德 3.5 索内特获得最高分 (0.84 ± 0.26),其次是聊天GPT-4o (0.76 ± 0.31).
- 克劳德 3.5 索内特报告了最高的置信度 (9.0 ± 0.9) 和表现出优越的一致性 (5.4%的响应变化).
- 所有聊天机器人在这些问题上都超过了以前的人类EDiR候选人.
结论:
- 克劳德3.5索内特在解决EDiR问题方面表现出卓越的准确性,信心和一致性.
- 聊天GPT-4o表现出强的表现,几乎与克劳德3.5索内特相匹配.
- 聊天机器人之间显著的性能差异需要在高风险环境中谨慎部署.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
575
06:37Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
4.1K
相关概念视频
Multiple Comparison Tests
4.0K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
4.0K
Non-equilibrium in the Cell
4.8K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
4.8K
Machines: Problem Solving II
367
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.
367
Machines: Problem Solving I
407
A toggle clamp is a mechanical device commonly used for holding and clamping objects in various applications, such as woodworking, metalworking, and assembly operations. Consider a toggle clamp subjected to a force of 200 N at the handle. The vertical clamping force can be calculated, provided the dimensions of the toggle clamp are known.
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
407
Distribution Reliability and Automation
155
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
155
Problem-Solving
238
Effective problem-solving consists of two steps: 1. identifying the problem and 2. selecting the appropriate problem-solving strategy (i.e., a plan of action used to find a solution). Humans use four problem-solving strategies:
238
