Validation of video-based skill assessment in carotid artery stenting

To develop weighted error-based, generic and procedure-specific rating scales, to validate these scales for video-based assessment during virtual carotid artery stent (CAS) procedures and correlate them with simulator-derived metrics.

A questionnaire was developed to assess the technique during live CAS procedures. Errors were rated from 1 (unimportant) to 5 (life-threatening) by 28 highly experienced CAS (>50 CAS) physicians. Virtual CAS procedure was performed by 21 interventionalists with varied CAS experience. Fluoroscopy screen and hand movements were video-taped, and simulator-derived metrics recorded. Experienced CAS practitioners then rated the video-taped performances using weighted error, generic and procedure-specific rating scales.

Of the 23 errors assessed, 12 were regarded as moderate (score 3), six serious (score 4) and four life-threatening (score 5). The generic rating scale was able to detect significant differences in performance between inexperienced and experienced CAS operators (score 25 vs. 32 respectively, P<0.01). All scoring systems demonstrated good inter-rater reliability (alpha=0.61-0.87). Significant correlations were observed between simulator-derived and video-based scores: weighted error-based score (r: 0.76, P<0.01), generic (r: 0.62, P<0.01) and procedure-specific (r: 0.76, P<0.01) rating scales.

The generic endovascular rating scale differentiated between levels of CAS experience among skilled interventionalists and correlated to simulator-based error scoring.