Evaluating Generative Artificial Intelligence as an Educational Tool for Radiology Resident Report Drafting.

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Radiology Education

Background:

• Radiology residents need timely, personalized feedback for skill development.
• Clinical workloads limit attending radiologists' ability to provide comprehensive guidance.
• Automated feedback systems can potentially bridge this gap in real-time.

Purpose of the Study:

• To evaluate a HIPAA-compliant GPT-4o system for automated feedback on resident breast imaging reports.
• To assess the accuracy and helpfulness of AI-generated feedback compared to attending radiologists.
• To determine the potential of AI as a scalable tool in radiology education.

Main Methods:

• Analysis of 5,000 resident-attending breast imaging report pairs.
• GPT-4o was prompted to identify common errors in resident reports.
• A reader study with 4 attending radiologists and 4 residents evaluated GPT-4o's feedback on 100 report pairs.

Main Results:

• GPT-4o demonstrated strong agreement with attending consensus on error identification (90.5%, 78.3%, 90.4%).
• AI feedback was rated as helpful in the majority of cases (89.8%, 83.0%, 92.0%).
• Replacing human readers with GPT-4o did not significantly alter inter-reader agreement.

Conclusions:

• GPT-4o reliably identifies key educational errors in radiology resident reports.
• The AI system shows potential as a scalable tool to enhance radiology education.
• Automated feedback can supplement traditional mentorship in radiology training.