视觉信息在多式联通大型语言模型性能中的作用:使用日本核医学考试委员会进行的评估
Takashi Watanabe1, Akira Baba2, Takeshi Fukuda2
1Department of Radiology, The Jikei University School of Medicine, 3-25-8 Nishi-Shimbashi, Minato-ku, Tokyo, 105-8461, Japan. bal.s.rimso@gmail.com.
最先进的多式联络大型语言模型 (LLM) 显示,对核医学考试的视觉数据的改进有限. 它们的诊断能力需要进一步提高,以便在医学成像分析中实际使用.
科学领域:
- 医疗成像中的人工智能
- 核医学 诊断 核医学 诊断
- 大型语言模型 (LLM)
背景情况:
- 多模式大语言模型 (LLM) 越来越多地被整合到各种领域,包括医学.
- 在专门的医学检查中评估他们的表现对于了解他们的潜力和局限性至关重要.
- 日本核医学委员会考试 (JNMBE) 是一个独特的挑战,因为它依赖于视觉诊断信息.
研究的目的:
- 在JNMBE问题上评估高级多式联络LLM (GPT-4o,Claude 3 Opus,Gemini 1.5 Pro) 的表现.
- 确定将视觉信息 (图像) 与纯文本数据相结合对LLM决策的影响.
- 在专业医疗环境中评估这些LLM的诊断准确性和响应一致性.
主要方法:
- 使用了92个JNMBE问题 (2019-2023) 附图片.
- 在两个条件下对LLM进行了测试:输入文本和图像,以及仅输入文本.
- 每个模型都回答了三次问题;最常见的答案被认为是最终的. 统计分析了准确度和一致率.
主要成果:
- 在GPT-4o和Claude 3 Opus的文字与图像和仅文本条件之间没有观察到显著的准确性差异.
- GPT-4o和Claude 3 Opus的图像准确率达到54.3%,但71.7%的答案符合只有文本的条件.
- 在文本和图像条件下,Gemini 1.5 Pro的表现明显低于GPT-4o;模型间的一致性为弱至中等.
结论:
- 目前的多式联络LLM在核医学委员会考试中显示,视觉信息的益处有限.
- 这些LLM在这个专业领域的诊断准确性目前不足以用于临床应用.
- 提高图像利用和响应可重现性对于未来将LLM纳入核医学教育和实践至关重要.
更多相关视频
09:27Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
Published on: October 13, 2018
08:05Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
Published on: June 30, 2020
相关概念视频
Imaging Studies II: Positron Emission Tomography and Scintigraphy
Fundamental Principles of PET
Higher Mental Functions of the Brain: Language
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Positron Emission Tomography
One of the main requirements of a PET scan is a positron-emitting radioisotope, which is produced in a cyclotron and then attached to a substance used by the part of the body...
Imaging Studies for Cardiovascular System III: X-Ray
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
Assessment of Airway, Skin Color, and Use of Accessory Muscles
Introduction
The initial evaluation of a patient's respiratory system...
Radiological Investigation II: MRI and Ventilation Perfusion Scan
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
