Related Experiment Video
Updated: Sep 10, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
681
Five advanced chatbots solving European Diploma in Radiology (EDiR) text-based questions: differences in performance
Jakub Pristoupil1, Laura Oleaga2, Vanesa Junquero3
1Department of Imaging Methods, Motol University Hospital and Second Faculty of Medicine, Charles University, Prague, Czech Republic.
European Radiology Experimental
|August 19, 2025
Summary
Claude 3.5 Sonnet led in performance, confidence, and consistency for European Diploma in Radiology (EDiR) questions. All tested AI chatbots surpassed human performance on these EDiR exam questions.
Area of Science:
- Artificial Intelligence in Medical Education
- Large Language Models in Radiology
- Chatbot Performance Evaluation
Background:
- Assessing the capabilities of AI-powered chatbots in medical examinations.
- Evaluating chatbot performance, confidence, and response consistency.
Purpose of the Study:
- To compare five large language model chatbots on their performance in solving European Diploma in Radiology (EDiR) multiple-response questions.
- To determine which chatbot exhibits superior accuracy, confidence, and response consistency.
Main Methods:
- Five chatbots (ChatGPT-4o, ChatGPT-4o-mini, Copilot, Gemini, Claude 3.5 Sonnet) were tested on 52 EDiR text-based multiple-response questions.
- Chatbots provided answers, confidence scores (0-10), and were evaluated over two iterations.
- A weighted formula calculated scores per question (0.0-1.0).
Main Results:
- Claude 3.5 Sonnet achieved the highest score (0.84 ± 0.26), followed by ChatGPT-4o (0.76 ± 0.31).
- Claude 3.5 Sonnet reported the highest confidence (9.0 ± 0.9) and demonstrated superior consistency (5.4% response variation).
- All chatbots outperformed previous human EDiR candidates on these questions.
Conclusions:
- Claude 3.5 Sonnet demonstrated superior accuracy, confidence, and consistency in solving EDiR questions.
- ChatGPT-4o showed strong performance, nearly matching Claude 3.5 Sonnet.
- Significant performance variations among chatbots necessitate cautious deployment in high-stakes settings.
Keywords:
Artificial intelligenceEducation (medical)Educational measurementEuropean Diploma in RadiologyRadiologyMore Related Videos
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
575
06:37Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
4.1K
Related Concept Videos
01:13Multiple Comparison Tests
4.0K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
4.0K
01:16Non-equilibrium in the Cell
4.8K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
4.8K
01:30Machines: Problem Solving II
367
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.
367
01:22Machines: Problem Solving I
407
A toggle clamp is a mechanical device commonly used for holding and clamping objects in various applications, such as woodworking, metalworking, and assembly operations. Consider a toggle clamp subjected to a force of 200 N at the handle. The vertical clamping force can be calculated, provided the dimensions of the toggle clamp are known.
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
407
01:25Distribution Reliability and Automation
155
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
155
01:29Problem-Solving
238
Effective problem-solving consists of two steps: 1. identifying the problem and 2. selecting the appropriate problem-solving strategy (i.e., a plan of action used to find a solution). Humans use four problem-solving strategies:
238