Integrating Video and Objective Performance Data: Development of VORTEX-SPV for Assessing Robotic Lobectomy Skills

Surgical education requires assessment tools as part of competency-based education (CBE). In robotic surgery, objective performance indicators (OPIs) are a novel objective method of measuring surgeon performance. We developed Video and Objective Robotic Task Evaluation indeX on Superior Pulmonary Vein dissection (VORTEX-SPV), combining video review and OPIs, to assess cardiothoracic (CT) fellow competency on SPV dissection during lobectomy.

CT fellows performed robotic lobectomy on an ex vivo perfused porcine model. Synchronized video and kinematic data were captured. Seven thoracic surgery attendings completed the robotic lobectomy to provide expert-level OPI data. VORTEX-SPV is a 12-item assessment with 4 performance levels including completion, safety, economy of motion, and optimized performance. We scored video-based review items (items 1 to 6) as 0 or 1 with an inter-rater agreement of 91%. We calculated OPIs from 6 categories (e.g., wrist articulation, instrument movement, energy, smoothness, clutching, instrument time) as within (1) or beyond (0) 2 standard deviations of the expert mean. We used Rasch modeling and item response theory to evaluate reliability and validity.

We assessed 2 cohorts of CT fellows (2019, n = 50; 2023, n = 59). There were 60% (65 of 109) who had complete data. Reliability was high (expected a posteriori = 0.90). Fourteen trainees scored up to the level of completion, 10 at safety, 17 at economy of motion, and 24 achieved optimized performance. All items appropriately increased in difficulty (Spearman's rho coefficient = 0.93).

VORTEX-SPV combines video review and objective performance data with strong reliability and validity evidence, serving as a robotic surgery assessment framework to facilitate CBE.