Related Experiment Video
Updated: Jan 9, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Performance of State-of-the-Art Multimodal Large Language Models on an Image-Rich Radiology Board Examination:
Takeshi Nakaura1, Naoki Kobayashi1, Takanori Masuda2
1Department of Central Radiology, Kumamoto University Hospital, 1-1-1 Honjo, Kumamoto 860-8556, Japan (T.N., N.K., Y.N., H.U., M.K., S.O., T.H.).
Rationale And Objectives:
This study aimed to assess the current multimodal capabilities of leading multimodal large language models (MLLMs) using a 2024 radiology board examination, evaluate their proficiency in utilizing medical image content, compare their performance against human examinees, and consider their cost-effectiveness.
Material And Methods:
Six contemporary MLLMs (GPT-4.1, o3, Claude 3.7 Sonnet, Claude 3.7 Sonnet-thinking, Gemini 2.5 Pro Preview, and Gemini 2.5 Flash Preview-thinking) were evaluated using the 100 multiple-choice questions (96 image-based) from the 2024 official board examination of the Japan Radiological Society. Questions, originally in Japanese, were instructed to be translated into English by the MLLMs. Performance was also analyzed with and without images for certain models to assess multimodal utility.
Results:
Gemini 2.5 Pro Preview achieved the highest accuracy (76.0%), followed by o3 (75.0%), both surpassing the average human examinee score (72.9%). Gemini 2.5 Pro Preview showed 75.0% accuracy with images versus 63.5% without (p = 0.035), and Gemini 2.5 Flash Preview-thinking demonstrated 68.8% accuracy with images versus 57.3% without (p = 0.019), indicating significant performance gains with image inclusion. Notably, Gemini models demonstrated top-tier performance at a highly competitive cost.
Conclusion:
The latest generation of MLLMs, particularly Gemini 2.5 Pro Preview and o3, can exceed average human performance on radiology board examinations and effectively leverage image information. The Gemini series, in particular, shows rapid improvements and offers a compelling combination of high performance and cost-efficiency for potential applications in radiology.
Summary Statement:
Modern multimodal large language models, notably Gemini 2.5 Pro Preview and o3, surpassed average human performance on the 2024 Japanese Radiology Board Examination. Gemini models showed significant score improvements when utilizing image data and offer top-tier performance at a competitive cost, indicating rapid advancements and excellent cost-effectiveness for radiology applications.
Related Concept Videos
Imaging Studies II: Positron Emission Tomography and Scintigraphy
Fundamental Principles of PET
Imaging Studies III: Computed Tomography
Imaging Studies IV: Magnetic Resonance Imaging
Imaging Studies I: CT and MRI
Description of the Procedures
Computed Tomography (CT) scan:
Computed Tomography (CT) scans use X-ray technology to generate detailed images of bones, organs, and tissues. During the scan, the patient lies on a moving table...
Imaging Studies for Cardiovascular System III: X-Ray
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
Imaging Studies VII: Vascular Imaging

