Video-based assessment for laparoscopic cholecystectomy: development of a rubric for summative evaluation of...

Operative skill is linked to patient outcomes, yet few procedure-specific assessment tools have sufficient validity evidence to guide competency decisions. To address this, the SAGES video-based assessment (VBA) task force is developing VBA tools across General Surgery subspecialties. This study applied a rigorous process to define the essential steps, errors, and skills needed to perform VBA for laparoscopic cholecystectomy (LC).

Following a literature review to identify key LC components, we conducted semi-structured interviews with subject matter experts (SMEs) to confirm procedural steps, identify critical errors, and clarify skills necessary for safe and effective performance. Interviews informed the development of a comprehensive task inventory questionnaire (TIQ), which was distributed to practicing surgeons performing LC who rated the criticality and difficulty of substeps, impact of errors, and importance and frequency of LC-specific skills using a 5-point Likert scale. Quantitative analysis of TIQ responses followed by application of three selection criteria confirmed the final rubric items while ensuring robust content validity evidence. Through iterative retreats, SMEs piloted the draft rubric on LC videos and refined the instrument.

SME interviewees (n = 6) included a mix of recently trained and experienced surgeons with expertise in hepatopancreatobiliary, acute care and/or minimally invasive surgery. A majority performed ≥ 50% of their LC cases with trainees and ≥ 25% for acute cholecystitis. Qualitative analysis of interview data yielded 4 main procedural steps, each with 1-5 substeps and 1-6 potential errors, as well as 8 LC-specific technical skills. TIQ respondents who perform > 20 LC/year were included (n = 645; 80% male, 63% with > 10 years LC experience, 52% North American) and had excellent overall agreement in their responses (α = 0.91). Quantitative analysis of TIQ data followed by application of three selection criteria identified 10 substeps nested within 3 key operative phases. Mean criticality and difficulty scores for these substeps were 4.35/5 and 3.19/5, respectively. Six procedural errors met inclusion criteria with a mean impact score of 4.40/5, and 6 technical skills were selected with mean current importance and frequency scores of 4.21/5 and 4.30/5, respectively. Assessment piloting prompted re-organization of rubric items under three modified steps to improve usability.

A comprehensive methodology was used to develop the SAGES VBA for LC while generating content validity evidence to support its use in summative assessment. Current efforts are focused on developing a rater training system and recruiting additional raters to facilitate reliability assessment and generate further validity evidence.