Novel evaluation of surgical activity recognition models using task-based efficiency metrics.

Area of Science:

• Robotic surgery
• Surgical training
• Machine learning in medicine

Background:

• Surgical training can be enhanced by focusing on specific tasks rather than entire procedures.
• Automated recognition of surgical activities using machine learning is crucial for efficient video review and feedback.
• Current models are typically evaluated using frame-level accuracy, which may not fully capture clinical utility.

Purpose of the Study:

• To propose and evaluate a novel metric-based approach for assessing surgical activity recognition models.
• To determine the adequacy of model performance for providing surgeon feedback based on task-specific metrics.
• To enable the quantification of surgical efficiencies through automated analysis.

Main Methods:

• Development of a CNN-LSTM model, RP-Net-V2, for recognizing 12 steps in robotic-assisted radical prostatectomies (RARP).
• Evaluation using conventional metrics like Jaccard Index and task boundary accuracy.
• Novel evaluation focusing on the accuracy of efficiency metrics derived from instrument movements and system events.

Main Results:

• The RP-Net-V2 model achieved a Jaccard Index of 0.85, surpassing previous models for RARP.
• Metrics derived from tasks automatically identified by RP-Net-V2 showed strong correlation with expert-labeled task metrics.
• Demonstrated the viability of using task-based efficiency metrics for model evaluation.

Conclusions:

• Metrics-based evaluation is a practical method for assessing surgical activity recognition models' readiness for quantifying surgical efficiencies.
• The proposed approach and findings highlight the potential for automated postoperative efficiency reporting.
• This facilitates focused training interventions to improve surgeon performance and patient outcomes.