Development and Validation of a Rubric for Evaluating Formative Feedback on Non-Technical Skills in General Surgery...

To create and validate a rubric to rate the quality of narrative feedback on technical and non-technical skills given to general surgery residents using a workplace-based assessment tool.

Instrument development and validation study using a focused literature review, modified Delphi process, stakeholder survey, and psychometric testing of narrative feedback comments from a mobile assessment application.

A single academic general surgery residency program using the System for Improving and Measuring Procedural Learning (SIMPL) application for operative assessment.

Panel of faculty and education specialists (modified Delphi); general surgery faculty and residents at the institution (stakeholder survey); and attendings who completed 1903 SIMPL evaluations between July 2020 and April 2024. A subset of 200 comments was double-coded for reliability and validity testing.

The final rubric included four domains: clarity, specificity, actionability, and tone. In the stakeholder survey (50 faculty, 29 residents), all domains were rated as important or very important for effective feedback, with a median rating of 5 on a 5-point Likert scale and no significant differences between groups. For 200 double-coded comments, intraclass correlation coefficients for nontechnical skill comments were 0.847 (clarity), 0.845 (specificity), 0.903 (actionability), and 0.950 (tone). For technical skill comments, coefficients were 0.733 (clarity), 0.790 (specificity), 0.763 (actionability), and 0.979 (tone). Cohen's kappa for classifying comment focus (technical, nontechnical, both, neither) was 0.808. Global usefulness ratings were strongly correlated with clarity (ρ = 0.775), specificity (ρ = 0.772), and actionability (ρ = 0.908; all p < 0.001), but not with tone.

This study presents a validated rubric that reliably rates the quality of narrative feedback on both technical and non-technical skills in operative assessment. The tool may support quality assurance, faculty development, and research focused on improving everyday feedback given to surgical residents.