Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

Area of Science:

• Surgical education
• Medical simulation
• Robotic surgery assessment

Background:

• Surgical trainees require feedback for skill development.
• Current feedback methods can be subjective and inefficient.
• Crowd-sourced assessment is explored as a novel feedback alternative.

Purpose of the Study:

• To compare crowd-sourced assessment with expert surgeon assessment for robot-assisted radical prostatectomy (RARP).
• To evaluate the reliability and validity of crowd workers' skill assessments in robotic surgery.

Main Methods:

• Utilized 45 video recordings of RARP modules from a robotic simulator.
• Employed crowd workers (CWs) and experienced RARP surgeons (ESs) for blinded video assessments.
• Used the modified Global Evaluative Assessment of Robotic Surgery (mGEARS) for evaluation.

Main Results:

• 149 CWs completed 1490 ratings; internal consistency was high (0.94).
• Inter-rater and test-retest reliability were low for CWs (0.29, 0.39) and moderate for ESs (0.61, 0.68).
• CWs could not differentiate surgeon skill levels (p=0.03-0.89), unlike ESs (p=0.034).

Conclusions:

• Significant disagreement exists between CW and ES assessments in RARP.
• CWs demonstrated inability to discern surgical expertise or suitability for patient care.
• Crowd-sourced assessment is not a viable substitute for expert evaluation in RARP training.