Validity of task-specific metrics for assessment in perineal proctectomy

Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics.

In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores.

In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ² = 15.38; P < 0.001; df = 2) and task-specific (χ² = 11.38; P = 0.003; df = 2) scores.

Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.