Video-based assessment of operative competency in endoscopic sinus surgery

The Accreditation Council for Graduate Medical Education requires that residency programs have an effective plan for assessing their residents' competence. Current evaluation tools used for the assessment of operative competency are subjective, recall based, and do not allow providing formative feedback to the resident. The purpose of this study was to evaluate the feasibility and validity of video-based assessment of endoscopic sinus surgery (ESS) skills.

Five junior (postgraduate year levels 1-3[PGY-1 to -3]) and three senior (PGY-4 and -5) residents were videotaped as they performed ESS on cadavers in the laboratory. Five faculty members watched the recorded videotapes and rated performance using global and checklist assessment tools for ESS. Evaluators were blinded to the identity of the performers. Rating was made feasible by allowing the raters to fast-forward through the tapes when appropriate. Construct validity was calculated by comparing total scores and score on each item of the instrument across the training levels. For all statistical purposes p < 0.05 was considered significant.

Results showed construct validity with senior residents performing better than junior residents. It took an average of 20 minutes (range, 7- 39 minutes) to watch and evaluate a video. Interrater reliability, as measured by interclass correlation coefficient across evaluators, was 0.62.

The use of videotapes combined with the objective assessment tool is a feasible and valid method for evaluating surgical skills. It has implications for residency programs, because it will increase faculty buy-in and reduce the potential for biased assessment.