Related Experiment Videos
Consensus operational definitions of surgical meta-competencies for computer vision-driven performance assessment: a
Alex H Lee1, Aviona Conti2, Matthew Leipzig3
1Department of Surgery, Stanford University, Stanford, CA, USA.
Background:
Operative video is increasingly used to evaluate surgical skills. However, existing assessment frameworks rely on subjective constructs or ordinal scales that are poorly suited for consistent annotation or computational analysis. Artificial intelligence (AI) and computer vision can enable automated and scalable performance assessment but require clearly defined, observable performance domains. We aimed to establish expert consensus on video-native surgical meta-competencies with definitions optimized for computer vision-driven assessment of operative performance.
Methods:
We conducted an international, two-round modified Delphi study involving surgeons with expertise in surgical education. Five candidate meta-competencies-tissue handling, psychomotor skills, progress, dissection quality, and exposure quality-were defined using binary adequacy-based criteria. Participants rated each definition on discrimination and operationality using five-point Likert scales. Definitions were iteratively refined until consensus (≥ 80%) was achieved.
Results:
Twenty experts from 14 institutions across seven countries participated (34% response rate). After Round 2, all five meta-competencies reached consensus, with mean discrimination scores ranging from 4.33 to 4.67 and operationality scores from 4.28 to 4.56. Progress demonstrated the highest discrimination, indicating strong ability to distinguish adequate from inadequate performance, whereas Dissection Quality demonstrated the highest operationality, suggesting that experts considered it the most readily assessable using video. Participants noted that visually identifiable features, such as effective tissue planes, together with binary adequacy criteria, improved confidence in applying the definitions during video review.
Conclusions:
We established consensus-based, video-native surgical meta-competencies that provide a standardized framework for computer vision-driven video review. By defining performance using observable constructs designed for consistent annotation across procedural hierarchies, this framework provides a starting point for scalable automated evaluation, pending empirical validation on operative video, to support surgical training and performance assessment.