Assessing the Capability of ChatGPT, Google Bard, and Microsoft Bing in Solving Radiology Case Vignettes

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology Education and Assessment
• Natural Language Processing in Healthcare

Background:

• Accurate medical image interpretation is crucial for radiology diagnosis and patient care.
• Advancements in AI and NLP prompt investigation into AI's role in assisting radiologists.
• Limited research exists on AI performance in radiology case interpretation versus human experts.

Purpose of the Study:

• To evaluate ChatGPT, Google Bard, and Bing performance on radiology case vignettes.
• To compare AI model responses against those of two radiology residents.
• To assess AI's capability in solving Fellowship of the Royal College of Radiologists 2A (FRCR2A) style questions.

Main Methods:

• 120 multiple-choice questions based on radiology case vignettes (FRCR2A style) were used.
• AI models (ChatGPT, Bard, Bing) and two radiology residents answered the questions.
• AI responses were compared to answer keys, with explanations rated by radiologists; a 60% passing score was set.

Main Results:

• Residents (63.33%, 57.5%) outperformed AI models (Bard 44.17%, Bing 53.33%, ChatGPT 45%); only one resident passed.
• Response patterns differed significantly (p=0.0117); AI models showed high agreement (ICC=0.628), while residents showed low agreement (Kappa=-0.376).
• AI-generated explanations for answers were only 44.72% accurate.

Conclusions:

• Human radiologists demonstrated superior accuracy and comprehension compared to the evaluated AI models.
• All tested AI models failed to achieve the passing score for the FRCR2A examination.
• AI models exhibited consistent responses, unlike residents, indicating potential for AI in standardized assessments but highlighting current limitations in nuanced interpretation.