Medical Students' Perception of Automated Note Feedback After Simulated Encounters

Grading medical student patient notes (PNs) is resource-intensive. Natural language processing (NLP) offers a promising solution to automatically grade PNs. We deployed an automated grading system that uses NLP and explored the perceived value of PN feedback.

The automated system graded written notes after two standardized patient encounters by third-year medical students. The system generated an individualized report on 'items found' and 'items not found' in the history, physical examination, and diagnosis sections, which was shared with students for feedback via a web-based interface. By rotation, block students received either the automated case feedback first or the faculty-written model note feedback first (the pre-intervention baseline).

After reviewing feedback, students completed surveys for both automated feedback and model note feedback and participated in follow-up focus groups. In total, 44 students received feedback, 37 completed surveys, and 28 participated in focus groups. Qualitative themes that emerged suggested the automated feedback was visually appealing and allowed for easy comparison of items found vs. missing, which would help improve students' documentation skills. Model note appeared trustworthy.

We found automated systems can be a potential tool for formative feedback on note writing activity although in terms of quality it does not surpass the pre-existing feedback methods, such as model note feedback used in our study. Order effects may have influenced these perceptions and the small sample size limits generalizability. Tested software had occasional errors in recognizing a phrase or showing a false positive.