Comparison of Large Language Models' Performance on 600 Nuclear Medicine Technology Board Examination-Style...

Area of Science:

• Nuclear Medicine
• Artificial Intelligence
• Medical Informatics

Background:

• Large language models (LLMs) are increasingly explored for medical applications.
• Retrieval-augmented generation (RAG) aims to enhance LLM accuracy by incorporating external knowledge.
• The utility of LLMs in specialized fields like nuclear medicine requires rigorous evaluation.

Purpose of the Study:

• To assess the performance of various LLMs, with and without RAG, in nuclear medicine.
• To compare the accuracy of leading LLMs across diverse nuclear medicine topics.
• To evaluate the potential of LLMs for professional education and clinical decision support in nuclear medicine.

Main Methods:

• Evaluated OpenAI GPT-4o series, Google Gemini, Cohere, and Meta Llama3 models.
• Tested models on 600 sample questions covering 15 nuclear medicine topics.
• Assessed accuracy with and without retrieval-augmented generation (RAG) implementation.

Main Results:

• OpenAI models (GPT-4o series) achieved the highest accuracy (0.787 with RAG).
• Anthropic Opus and Google Gemini 1.5 Pro also showed strong performance with RAG.
• LLMs demonstrated improved accuracy with RAG, particularly in radiation safety and skeletal scintigraphy.

Conclusions:

• LLMs, especially with RAG, offer significant potential for enhancing nuclear medicine education and decision-making.
• While accuracy is promising, challenges persist in interpreting complex guidelines and visual data.
• Further optimization of LLMs is crucial for their reliable integration into clinical nuclear medicine practice.