Natural Language Processing and Assessment of Resident Feedback Quality

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Surgical Training

Background:

• Surgical residency programs require high-quality feedback for trainee development.
• Manual assessment of narrative feedback is time-consuming and subjective.
• Objective methods are needed to evaluate feedback quality efficiently.

Purpose of the Study:

• To validate a natural language processing (NLP) model for characterizing surgical trainee feedback quality.
• To assess the accuracy, sensitivity, specificity, and AUROC of the NLP model.

Main Methods:

• Collected narrative feedback transcripts from a surgical residency program.
• Trained a logistic regression NLP model using 75% of classified transcripts.
• Tested the NLP model on 25% of unclassified transcripts, classifying them as high- or low-quality.

Main Results:

• The NLP model achieved 83% accuracy in classifying feedback quality.
• The model demonstrated high specificity (97%) and an AUROC of 0.86.
• Sensitivity was 37%, indicating room for improvement in identifying all low-quality feedback.

Conclusions:

• NLP models can accurately and specifically classify the quality of operative performance feedback.
• NLP provides an efficient tool for residency programs to measure feedback quality.
• This technology can support feedback improvement initiatives and enhance surgical trainee education.