ChatGPT performance on radiation technologist and therapist entry to practice exams

Area of Science:

• Artificial Intelligence in Medical Education
• Large Language Models (LLMs) for Professional Certification
• Medical Imaging and Radiation Therapy Assessment

Background:

• Evaluating the capabilities of advanced AI like ChatGPT (GPT-4) is crucial for understanding its potential role in professional development.
• The Canadian Association of Medical Radiation Technologists (CAMRT) offers certification exams that serve as a benchmark for professional proficiency.

Purpose of the Study:

• To assess the proficiency of ChatGPT (GPT-4) on certification-style examinations administered by the CAMRT.
• To analyze ChatGPT's performance consistency across multiple attempts on these standardized exams.

Main Methods:

• ChatGPT (GPT-4) was administered practice exams from CAMRT across radiological technology, MRI, nuclear medicine, and radiation therapy.
• Each exam was attempted five times, with performance analyzed using descriptive statistics and stratified by discipline and question type (knowledge, application, critical thinking).
• Light's Kappa statistic was employed to measure the agreement of answers across repeated attempts.

Main Results:

• ChatGPT achieved passing scores (≥65%) on MRI (100%) and radiation therapy (100%) exams across all attempts.
• Performance varied for radiological technology (20% pass rate) and nuclear medicine (60% pass rate).
• The AI excelled in knowledge-based questions but performed poorly on critical thinking questions; response agreement across attempts was substantial to almost perfect.

Conclusions:

• ChatGPT (GPT-4) can pass certification-style exams for radiation technologists and therapists, though performance differs by discipline.
• The AI exhibits high consistency in its responses across repeated exam attempts.
• Future research on AI performance in standardized testing should incorporate repeated measures to account for response consistency.