Differentiating between GPT-generated and human-written feedback for radiology residents

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Radiology Training

Background:

• Competency-based medical education (CBME) in Canadian radiology programs necessitates increased faculty assessment.
• The proliferation of large language models (LLMs) prompts an evaluation of their utility in generating narrative feedback for resident assessments.

Purpose of the Study:

• To compare human-written feedback with GPT-3.5-generated feedback for radiology residents.
• To assess the ability of human raters and GPT-3.5 to differentiate between human and AI-generated feedback.

Main Methods:

• 110 resident feedback comments from a Canadian Diagnostic Radiology program were analyzed.
• 110 synthetic comments were generated using GPT-3.5 and mixed with actual human comments.
• Two faculty raters and GPT-3.5 attempted to identify the source of each comment.

Main Results:

• Human feedback was generally longer and more specific than GPT-3.5 generated comments.
• Differentiation between human and AI feedback was harder when comments were vague.
• Human raters achieved 80.5% accuracy in source identification, while GPT-3.5 achieved only 50%.

Conclusions:

• GPT-3.5 currently falls short of human expert capabilities in providing specific, nuanced feedback for radiology residents.
• GPT-3.5 demonstrates a lower capacity than humans in distinguishing between authentic and synthetic feedback.
• Findings may inform the development of advanced AI algorithms for improved feedback generation and faculty development support.