Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication

Automated assessment of surgical skills using artificial intelligence (AI) is valuable for trainees to obtain instantaneous feedback. After bimanual tool motions are captured, the derived kinematic metrics have shown to be reliable predictors of performance in laparoscopic tasks. Implementing automated tool tracking assessment requires time-intensive human annotation. We have developed AI-based tool tracking using the Segment Anything Model (SAM) to eliminate the need for human annotators. Here we describe a study to evaluate the usefulness of our tool tracking model in automated assessment of performance in a laparoscopic suturing task of the fundoplication procedure.

An automated tool tracking model was applied to recorded videos of Nissen fundoplication on porcine bowel. The participating surgeons were grouped into novices (PGY1-2) and experts (PGY3-5, fellow and attendings). The beginning and the ending of each of the suturing steps were segmented and the motions of the left and right tools were extracted. A low-pass filter with a cut-off frequency of 24 Hz was then applied to remove any noise. Automated assessment of performance was implemented using both supervised and unsupervised models, and an ablation study was performed to compare the performance. For the supervised learning model, kinematic features included root mean square (RMS) velocity, RMS acceleration, RMS jerk, total path length, and Bimanual Dexterity in pixel coordinates (x, y) that were extracted and analyzed using Logistic Regression, Random Forest Classifier, Support Vector Classifier, and XGBoost. We further analyzed the performance with a reduced set of features selected using a Principal Component Analysis (PCA). For unsupervised learning, a Denoising Autoencoder (DAE) model with classifiers like a 1-D (Convolutional Neural Network) CNN and traditional Machine Learning models were trained for classification.

Data was extracted for 28 participants out of an initial cohort of 38, categorized into 9 novices and 19 experts. The approach for supervised learning utilized kinematic features with Principal Component Analysis (PCA) using a Random Forest model (the best model among other Machine Learning models) and obtained an accuracy of 0.795 ± 0.065 and an F1 score of 0.778 ± 0.071. The approach for unsupervised learning employed a 1-D CNN achieved the best results with an accuracy of 0.817 ± 0.108 and an F1 score of 0.806 ± 0.110. This second approach is superior as it eliminates the need to compute kinematic features.

We successfully demonstrated an AI model for automated classification of performance independent of human annotation of surgical videos that can be used to enhance surgical skill evaluation.