Comparative Analysis of LLMs' Performance On a Practice Radiography Certification Exam

Area of Science:

• Artificial Intelligence in Medical Education
• Radiography Certification Examination
• Natural Language Processing

Background:

• The increasing integration of artificial intelligence (AI) and large language models (LLMs) into various professional fields necessitates an evaluation of their performance in specialized domains.
• Assessing the capabilities of LLMs in medical education, particularly for high-stakes certification exams, is crucial for understanding their potential and limitations.

Purpose of the Study:

• To compare the performance of five leading large language models (LLMs) on a practice radiography certification examination.
• To identify which LLM exhibits the highest accuracy and to analyze performance across different content categories.

Main Methods:

• An exploratory, nonexperimental study was conducted using 200 multiple-choice questions from a practice radiography certification exam.
• Five LLMs—ChatGPT, Claude, Copilot, Gemini, and Perplexity—were evaluated based on their accuracy in answering these questions.
• Statistical analysis, including McNemar tests, was employed to determine significant differences in accuracy rates, with performance also broken down by content categories and subcategories.

Main Results:

• ChatGPT achieved the highest overall accuracy at 83.5%, outperforming Claude and Gemini significantly.
• ChatGPT excelled in patient care questions, answering all correctly, and showed strong performance in safety and procedures.
• Copilot demonstrated the highest accuracy in image production, while ChatGPT led in four out of eight subcategories.

Conclusions:

• ChatGPT was the most accurate large language model for the assessed practice radiography certification exam questions.
• Despite superior performance, users must exercise caution with generative AI, verifying all responses for accuracy due to the potential for generating incorrect information.
• Future research should explore LLM performance on certification exams in other medical imaging modalities like MRI, nuclear medicine, radiation therapy, and sonography.