Ambient AI Scribes to Create Educational Feedback Notes for Medical Students: A Randomized Trial

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Clinical Competence Development

Background:

• High-quality observation and feedback are crucial for medical trainees' clinical competence and professional growth.
• Faculty face challenges in providing timely written feedback due to documentation burden and competing demands.
• Ambient AI scribes offer a potential solution by capturing verbal exchanges and generating structured clinical notes.

Purpose of the Study:

• To evaluate the efficacy of ambient AI scribes in generating educational feedback notes.
• To assess AI scribe impact during a formative medical interviewing workshop for first-year medical students.

Main Methods:

• Thirteen instructors were randomized into control (human-only) and intervention (AI scribe-assisted) groups.
• AI scribe group used AI to transcribe and summarize student-instructor encounters into feedback notes, followed by instructor editing.
• Feedback quality (EFeCT), task load (NASA-TLX), and usability (SUS) were assessed; factual accuracy of AI summaries was reviewed.

Main Results:

• AI scribe-assisted feedback (human-edited and unedited) received significantly higher quality scores (median 3.00) than human-only feedback (median 2.00).
• AI-assisted outputs were longer than human-only narratives, with a 6.8% mischaracterization and 1.7% hallucination rate in unedited summaries, largely corrected by instructors.
• Task load and usability were comparable between groups, indicating no significant increase in instructor effort with AI assistance.

Conclusions:

• Ambient AI scribe-assisted workflows enhance the quality of written narrative feedback in medical education.
• This approach offers a potential solution to documentation burden without increasing instructor workload.
• AI scribes represent a promising innovation for improving feedback documentation in medical training.