Utilizing a novel tool for the comprehensive assessment of resident operative performance

A mechanism for more effective and comprehensive assessment of surgical residents' performance in the operating room (OR) is needed, especially in light of the new requirements issued by the American Board of Surgery. Furthermore, there is an increased awareness that assessments need to be more meaningful by including not only procedure-specific and general technical skills, but also nontechnical skills (NOTECHS), such as teamwork and communication skills. Our aims were to develop a methodology and create a tool that comprehensively assesses residents' operative performance.

A procedure-specific technical skill assessment for laparoscopic colon resections was created through use of task analysis. Components of previously validated tools were added to broaden the assessment to include general technical skills and NOTECHS. Our instrument was then piloted in the OR to measure face and content validity through an iterative process with faculty evaluators. Once the tool was finalized, postgraduate 3 (PG3) and PG5 residents on a 2-month long rotation were assessed by 1 of 4 colorectal surgeons immediately after completing a case together. Construct validity was measured by evaluating the difference in scores between PG3 and PG5 residents' performance as well as the change in scores over the course of the rotation.

Sixty-three assessments were performed. All evaluations were completed within 48 hours of the operation. There was a statistically significant difference between the PG3 and PG5 scores on procedure-specific performance, general technical skills, NOTECHS, and overall performance. Over the course of the rotation, a statistically significant improvement was found in residents' scores on the procedure-specific portion of the assessment but not on the general surgical skills or NOTECHS.

This is a feasible, valid, and reliable assessment tool for the comprehensive evaluation of resident performance in the OR. We plan to use this tool to assess resident operative skill development and to improve direct resident feedback.