多选择题在生成人工智能和医学教育中的陷
Shrutika Singh1, Anton Alyakin2,3,4, Daniel Alexander Alber2,5
1Department of Neurosurgery, NYU Langone Health, 550 First Ave, New York, NY, 10016, USA. ss17775@nyu.edu.
Scientific reports
|November 26, 2025
概括
大型语言模型 (LLM) 可能会因为多选题 (MCQ) 格式而高估医疗知识. 自由答案问题揭示了显著的性能差距,表明当前的医疗AI基准有缺陷.
科学领域:
- 人工智能的人工智能
- 医学教育 医学教育
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 越来越多地在医学多选题 (MCQ) 基准上进行评估.
- 这种表现往往被认为是他们医学专业知识和推理能力的证据.
研究的目的:
- 调查医学MCQ上的LLM绩效是否被与真正的医学知识无关的因素所膨胀.
- 开发和利用一个新的基准 (FreeMedQA) 来比较免费响应与MCQ医疗问题的LLM绩效.
主要方法:
- 创建了FreeMedQA,这是一个具有配对自由响应和MCQ医疗问题的基准.
- 在FreeMedQA.上评估了三个高级LLM (GPT-4o,GPT-3.5,LLama-3-70B指导)
- 进行了一项掩盖研究,以分离问题格式对LLM绩效的影响.
主要成果:
- 与MCQ相比,LLM在自由答案问题上显示了39.43%的显著平均绩效下降 (p = 1.3 * 10-5),超过了人类22.29%的下降.
- 在一个掩盖研究中,LLM MCQ的表现甚至在100%掩盖的情况下也保持在机会之上,而自由响应的表现接近于零.
- 在100%掩盖的MCQ条件下,GPT-4o实现了37.34%的准确性.
结论:
- 目前的医疗MCQ基准可能会高估医学LLM的能力.
- 通过LLM评估的自由反应评估,提供了更准确的医学知识和推理的衡量标准.
- 这项研究表明,人类和机器医疗评估的改进.
相关概念视频
Documentation of Nursing Diagnosis
1.6K
The nurse documents nursing diagnoses and enters them into the patient record. The identified patient's nursing diagnosis is either written out with a plan of care or entered into the electronic health record.
In some settings, data-driven computerized decision support systems are in place, allowing for more accurate nursing diagnoses. The database within one of these systems includes diagnostic labels defining characteristics, activities, and indicators for nursing. A nurse enters...
In some settings, data-driven computerized decision support systems are in place, allowing for more accurate nursing diagnoses. The database within one of these systems includes diagnostic labels defining characteristics, activities, and indicators for nursing. A nurse enters...
1.6K
The Availability Heuristic
6.9K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
6.9K
Hindsight Biases
4.2K
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
4.2K
Systematic Error: Methodological and Sampling Errors
8.5K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
8.5K
Multi-input and Multi-variable systems
376
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...
376
Multiple Comparison Tests
4.4K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
4.4K


