在日本放射学委员会考试问题上评估推理大语言模型的性能
Takeshi Nakaura1, Hiroto Takamure1, Naoki Kobayashi1
1Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Chuo-ku, Kumamoto-shi, Kumamoto 860-8556, Japan (T.N., H.T., N.K., K.S., N.Y., Y.N., H.U., M.K., T.H.).
与基础模型相比,新的推理大语言模型 (LLM) 在日本放射学委员会考试中表现优越. 然而,这些先进的LLM需要更高的成本和更长的处理时间.
科学领域:
- 医疗成像中的人工智能
- 放射学教育和评估 放射学教育和评估
- 在医疗保健中的自然语言处理.
背景情况:
- 大型语言模型 (LLM) 在医学教育和评估中的整合正在迅速发展.
- 评估专业的LLM对日本放射学委员会等高风险的检查的有效性至关重要.
研究的目的:
- 评估OpenAI的推理LLM (o1-预览,o1-迷你) 的性能,成本和处理时间,与其基础模型 (GPT-4o,GPT-4o-迷你) 相比.
- 确定这些LLM在回答日本放射学委员会各个子专业的考试问题时的准确性.
主要方法:
- 使用了210个日本放射学委员会考试问题 (2022-2023) 的数据集.
- 四个LLM (o1-预览,o1-mini,GPT-4o,GPT-4o-mini) 在问题准确性,成本和处理时间方面进行了评估.
- 应用麦克纳马测试来比较精度差异的统计学意义.
主要成果:
- o1-预览模型实现了最高准确度 (85.7%),显著超过了GPT-4o (73.3%).
- o1-mini模型 (69.5%) 的表现也明显超过了GPT-4o-mini (46.7%).
- 与基准模型相比,推理模型的成本更高,处理时间更长.
结论:
- 推理LLM在放射学委员会考试问题上表现显著提高.
- 提高LLM推理的准确性是以增加计算成本和处理时间为代价的.
- 这些发现强调了在应用先进的LLM到专业医疗评估时,性能和资源利用之间的权衡.
更多相关视频
05:33Experimental Paradigm for Measuring the Effect of Induced Emotion on Grammar Learning
Published on: January 29, 2020
09:00Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
Published on: August 16, 2024
相关概念视频
Deductive Reasoning
For example, a researcher can deduce specific predictions...
Inductive Reasoning
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Reasoning
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
Positron Emission Tomography
One of the main requirements of a PET scan is a positron-emitting radioisotope, which is produced in a cyclotron and then attached to a substance used by the part of the body...
Radiological Investigation III: Pulmonary Angiogram and PET Scan
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Critical Thinking II
