Using large language model to aid in teaching medical imaging report writing

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Radiology Reporting

Background:

• Medical education increasingly utilizes AI tools.
• Standardization and accuracy in radiology reports are critical for patient care.
• Large Language Models (LLMs) offer potential for educational feedback.

Purpose of the Study:

• Compare free Large Language Models (LLMs) for feedback effectiveness.
• Assess LLM-generated feedback's impact on student imaging report quality.
• Investigate LLM feedback's role in improving report accuracy and standardization.

Main Methods:

• Evaluated feedback quality from four free LLMs.
• Assessed student imaging reports using a residency training scale.
• Compared a test group receiving LLM feedback with a control group.
• Surveyed student satisfaction with LLM feedback.

Main Results:

• ChatGPT 3.5, ERNIE Bot v3.5, and Tongyi v2.5 provided superior feedback structure and logic compared to Claude 3 OPUS.
• All LLMs exhibited some degree of hallucination.
• Student reports significantly improved in the test group compared to the control group.

Conclusions:

• LLM feedback aids students in critically evaluating and improving imaging reports.
• LLM-generated feedback can enhance reporting skills.
• Teacher supervision is essential when using LLMs for feedback in medical education.