在日本诊断放射学委员会考试 (2021-2023) 中多式大型语言模型的性能 (2021-2023)
Takeshi Nakaura1, Naofumi Yoshida1, Naoki Kobayashi1
1Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
Academic radiology
|November 9, 2024
概括
多模式大语言模型 (LLM) 在日本诊断放射学委员会考试中显示出有限的准确性,性能因模型和子专业而异. 添加图像并没有提高LLM的准确性,凸显了进一步开发的需要.
科学领域:
- 医疗成像中的人工智能
- 放射学委员会考试 放射学委员会考试
- 大型语言模型
背景情况:
- 多模式大语言模型 (LLM) 是新兴的工具,在医学诊断中具有潜在的应用.
- 在专门的医疗检查中评估这些先进的人工智能模型的性能对于了解它们目前的能力至关重要.
研究的目的:
- 评估在日本诊断放射学委员会考试 (JDRBE) 上领先的多式联络LLM的准确性.
- 为了比较模型的性能,并没有包括医疗图像.
- 为了识别不同放射学子专业的性能差异.
主要方法:
- 评估了五个多模式的LLM (GPT-4o,Claude 3 Opus,GPT-4 Vision,Gemini Flash 1.5,Gemini Pro 1.5) 并进行了评估.
- 模型使用2021-2023年的JDRBE问题进行了测试.
- 通过总体准确率和分专业准确率来衡量性能,在有图像和没有图像的条件下.
主要成果:
- 平均LLM准确度在30.21%至45.00%之间,GPT-4o获得最高分数 (45.00%).
- 没有图像的Claude 3 Opus表现最好 (45.83%).
- 包括图像并没有显著提高任何模型的准确性,没有一个模型超过了60%的合格门.
结论:
- 多模式LLM在JDRBE上显示了可变的性能,其中GPT-4o和Claude 3 Opus显示了最高的精度.
- 目前用于评估放射性图像的多模式LLM能力有限,需要为常规临床使用进行进一步的研究和开发.
- 各个子专业之间的性能差异以及图像集成方面的缺乏改进表明了未来模型增强的领域.
相关概念视频
01:25Imaging Studies II: Positron Emission Tomography and Scintigraphy
88
Positron Emission Tomography (PET) is a medical imaging technique that provides crucial insights into the body's physiological functions at a molecular level. It is an indispensable resource for diagnosing, staging, and monitoring various illnesses, notably cancer, neurological disorders, and cardiovascular conditions.
Fundamental Principles of PET
Fundamental Principles of PET
88
01:10Higher Mental Functions of the Brain: Language
744
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
744
01:30Radiological Investigation II: MRI and Ventilation Perfusion Scan
100
Description
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
100
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
86
Radiological investigations are paramount in the diagnosis and management of various pulmonary diseases. Two essential investigations are the Pulmonary Angiogram and the Positron Emission Tomography (PET) Scan.
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
86
01:20Imaging Studies for Cardiovascular System III: X-Ray
145
The most common cardiovascular diagnostic test is an X-ray. It produces images of the heart, blood vessels, and adjacent structures.
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
145
01:14Imaging Studies I: CT and MRI
197
Introduction: MRI and CT scans are crucial advancements in medical imaging techniques, playing a vital role in diagnosing conditions related to the gastrointestinal (GI) system. Each scan serves distinct purposes, targets specific areas, and requires unique nursing duties.
Description of the Procedures
Computed Tomography (CT) scan:
Computed Tomography (CT) scans use X-ray technology to generate detailed images of bones, organs, and tissues. During the scan, the patient lies on a moving table...
Description of the Procedures
Computed Tomography (CT) scan:
Computed Tomography (CT) scans use X-ray technology to generate detailed images of bones, organs, and tissues. During the scan, the patient lies on a moving table...
197