Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam

Area of Science:

• Medical Physics
• Artificial Intelligence
• Radiological Protection

Background:

• Generative Pre-Trained Transformers (GPT) are advanced AI language models.
• Their application in specialized scientific fields like health physics requires rigorous evaluation.
• Assessing AI performance in high-stakes domains is crucial for safe implementation.

Purpose of the Study:

• To evaluate the efficacy of OpenAI's GPT-3.5 and GPT-4 models in radiological protection and health physics.
• To determine if these AI models can accurately answer questions simulating a health physics certification exam.

Main Methods:

• A set of 1064 surrogate questions, mirroring a health physics certification exam, was used.
• GPT-3.5 and GPT-4 models were tested using a standardized, simple prompting strategy.
• Performance was assessed across five distinct knowledge domains within health physics.

Main Results:

• Neither GPT-3.5 (45.3% weighted average) nor GPT-4 (61.7% weighted average) met the 67% passing threshold.
• GPT-4 demonstrated superior accuracy across all tested domains compared to GPT-3.5.
• GPT-3.5 exhibited better answer formatting, while GPT-4 showed higher overall correctness.

Conclusions:

• Current GPT models, including GPT-4, are not sufficiently accurate for independent use in radiological protection certification.
• While promising for domain-specific content, caution is advised for AI deployment in health physics.
• Human oversight and verification remain essential for AI applications in this critical field.