Automated assessment of non-technical skills by heart-rate data

Observer-based scoring systems, or automatic methods, based on features or kinematic data analysis, are used to perform surgical skill assessments. These methods have several limitations, observer-based ones are subjective, and the automatic ones mainly focus on technical skills or use data strongly related to technical skills to assess non-technical skills. In this study, we are exploring the use of heart-rate data, a non-technical-related data, to predict values of an observer-based scoring system thanks to random forest regressors.

Heart-rate data from 35 junior resident orthopedic surgeons were collected during the evaluation of a meniscectomy performed on a bench-top simulator. Each participant has been evaluated by two assessors using the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score. A preprocessing stage on heart-rate data, composed of threshold filtering and a detrending method, was considered before extracting 41 features. Then a random forest regressor has been optimized thanks to a randomized search cross-validation strategy to predict each score component.

The prediction of the partially non-technical-related components presents promising results, with the best result obtained for the safety component with a mean absolute error of 0.24, which represents a mean absolute percentage error of 5.76%. The analysis of feature important allowed us to determine which features are the more related to each ASSET component, and therefore determine the underlying impact of the sympathetic and parasympathetic nervous systems.

In this preliminary work, a random forest regressor train on feature extract from heart-rate data could be used for automatic skill assessment and more especially for the partially non-technical-related components. Combined with more traditional data, such as kinematic data, it could help to perform accurate automatic skill assessment.