Evaluating ChatGPT-4's Performance in Identifying Radiological Anatomy in FRCR Part 1 Examination Questions

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Education

Background:

• Accurate radiological image interpretation is crucial for patient care.
• Artificial intelligence (AI) and natural language processing (NLP) show potential for supporting radiologists.
• Research on AI performance in radiology is still developing.

Purpose of the Study:

• To evaluate ChatGPT-4's effectiveness in answering radiological anatomy questions.
• To assess performance on questions similar to the Fellowship of the Royal College of Radiologists (FRCR) Part 1 Anatomy examination.

Main Methods:

• 100 mock FRCR Part 1 Anatomy questions were used.
• ChatGPT-4 was tested with and without examination context.
• Responses were scored for correctness, sidedness, modality identification, and approximation.

Main Results:

• ChatGPT-4 achieved low correctness scores (4-7.5%) but 100% accuracy in identifying imaging modalities.
• The model struggled with sidedness (40-42%) and showed limited response consistency (32%).
• Approximation scores exceeded 50%, indicating identification of structures not directly queried.

Conclusions:

• ChatGPT-4 demonstrates significant limitations in interpreting normal radiological anatomy.
• Enhanced AI training is necessary for accurate anatomical interpretation.
• Identifying the correct side of anatomical structures remains a challenge for current AI models.