Assessing knowledge about medical physics in language-generative AI with large language model: using the medical...

Area of Science:

• Medical Physics
• Artificial Intelligence
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in specialized fields.
• Evaluating AI performance on professional certification exams is crucial for understanding capabilities.

Purpose of the Study:

• To assess the performance of ChatGPT-3.5 and ChatGPT-4.0 on the Japanese medical physicist board examinations.
• To establish a benchmark for medical physics knowledge within these AI models.

Main Methods:

• Utilized examination questions from 2018-2022 Japanese medical physicist board exams.
• Compared AI-generated answers from ChatGPT-3.5 and ChatGPT-4.0 against correct answers.
• Analyzed accuracy rates across different medical physics categories.

Main Results:

• ChatGPT-4 achieved an average accuracy of 72.7% ± 2.6%, significantly higher than ChatGPT-3.5's 42.2% ± 2.5%.
• Accuracy exceeded 60% in most categories for ChatGPT-4, but remained below 60% in radiation metrology (55.6%) and radiation-related laws/ethics (40.0%).

Conclusions:

• ChatGPT-4 demonstrates superior performance in medical physics knowledge compared to ChatGPT-3.5.
• Current LLMs require further development for high accuracy in all specialized medical physics domains.
• These findings provide foundational data for developing AI-driven medical physics tools, such as Japanese-input radiation therapy support systems.