Large Language Model Cost and Performance: A Comprehensive Analysis in the Context of the Japan Radiology Board...

Area of Science:

• Artificial Intelligence in Medical Education
• Natural Language Processing in Radiology
• Machine Learning for Board Examinations

Background:

• The Japan Radiology Board Examination (JRBE) assesses essential knowledge for radiologists.
• Evaluating the efficacy of Large Language Models (LLMs) in specialized medical examinations is crucial.
• Understanding LLM performance variations based on model type (open-source vs. closed-source) and language is key.

Purpose of the Study:

• To assess the effectiveness of various LLMs in answering JRBE questions.
• To compare the performance of open-source and closed-source LLMs on the JRBE.
• To investigate the impact of language (Japanese vs. English translation) on LLM performance in this context.

Main Methods:

• Administered 315 JRBE questions (2021-2023) to 14 LLMs (7 open-source, 7 closed-source).
• Evaluated models using original Japanese questions and LLM-translated English versions.
• Analyzed performance metrics including median scores, IQR, P-values, and correlation coefficients using Python.

Main Results:

• Closed-source LLMs achieved a 50.2% higher correct response rate than open-source models (44.28% vs. 29.52%, P < 0.001).
• Translating questions to English improved median scores by 21.1% (33.02% to 40.00%, P = 0.005).
• Higher LLM cost correlated positively with accuracy (r = 0.613-0.623, P < 0.020); release date did not significantly impact performance.

Conclusions:

• High-end, closed-source LLMs demonstrate superior performance on the JRBE.
• English translation of questions by LLMs enhances accuracy for these models.
• LLM cost is a significant predictor of performance, while model age is not.