Large Language Model Cost and Performance: A Comprehensive Analysis in the Context of the Japan Radiology Board...

This study aims to evaluate various large language models (LLMs) for their effectiveness in answering Japan Radiology Board Examination (JRBE).

A total of 315 examination questions from 2021 to 2023 JRBE were administered to 14 LLMs, comprising 7 open-source and 7 closed-source models. Each model processed the questions in their original Japanese language and after translating them into English by LLMs themselves. Performance metrics, including median scores, interquartile ranges (IQR), P -values, and correlation coefficients, were analyzed using Python.

Closed-source models achieved a higher median correct response rate of 44.28% (IQR: 32.38%-53.02%) compared with open-source models at 29.52% (IQR: 24.76%-36.67%), exhibiting a 50.2% improvement ( P < 0.001). Translating questions to English improved performance, with median scores increasing from 33.02% (IQR: 27.86%-43.65%) to 40.00% (IQR: 27.54%-46.27%), representing a 21.1% increase ( P = 0.005). A positive correlation was observed between the cost per 1M tokens and accuracy (Japanese: r = 0.623, P = 0.017; English: r = 0.613, P = 0.020). No significant correlation was found between model release dates and performance. Only the high-cost closed-source models, GPT-4 and Claude-3-Opus, surpassed the 60% passing threshold by Japanese Medical Specialty Board when using English translation. Among open-source models, LLama-3-70B demonstrated notable performance improvements compared with its predecessors.

Closed-source, high-end LLMs exhibit superior performance in JRBE, and translating questions into English by themselves further enhances their accuracy. There is a significant positive correlation between the cost of LLMs and their performance, whereas the release date does not significantly influence their performance.