Related Experiment Video
Updated: Feb 28, 2026
The Immersive Cleveland Clinic Virtual Reality Shopping Platform for the Assessment of Instrumental Activities of Daily Living
Published on: July 28, 2022
Development and validation of a tool for non-technical skills evaluation in robotic surgery-the ICARS system
Nicholas Raison1,2, Thomas Wood1, Oliver Brunckhorst1
1Division of Transplantation Immunology & Mucosal Biology, Faculty of Life Sciences & Medicine, Guy's Hospital, MRC Centre for Transplantation, King's College London, 5th Floor Tower Wing, London, SE1 9RT, UK.
Background:
Non-technical skills (NTS) are being increasingly recognised as vital for safe surgical practice. Numerous NTS rating systems have been developed to support effective training and assessment. Yet despite the additional challenges posed by robotic surgery, no NTS rating systems have been developed for this unique surgical environment. This study reports the development and validation of the first NTS behavioural rating system for robotic surgery.
Methods:
A comprehensive index of all relevant NTS behaviours in robotic surgery was developed through observation of robotic theatre and interviews with robotic surgeons. Using a Delphi methodology, a panel of 16 expert surgeons was consulted to identify behaviours important to NTS assessment. These behaviours were organised into an appropriate assessment template. Experts were consulted on the feasibility, applicability and educational impact of ICARS. An observational trial was used to validate ICARS. 73 novice, intermediate and expert robotic surgeons completed a urethrovesical anastomosis within a simulated operating room. NTS were tested using four scripted scenarios of increasing difficulty. Performances were video recorded. Robotic and NTS experts assessed the videos post hoc using ICARS and the standard behavioural rating system, NOn-Technical Skills for Surgeons (NOTSS).
Results:
28 key non-technical behaviours were identified by the expert panel. The finalised behavioural rating system was organised into four principle domains and seven categories. Expert opinion strongly supported its implementation. ICARS was found to be equivalent to NOTSS on Bland-Altman analysis and accurately differentiated between novice, intermediate and expert participants, p = 0.01. Moderate agreement was found between raters, Krippendorff's alpha = 0.4. The internal structure of ICARS was shown to be consistent and reliable (median Cronbach alpha = 0.92, range 0.85-0.94).
Conclusion:
ICARS is the first NTS behavioural rating system developed for robotic surgery. Initial validation has shown it to be an effective and reliable tool. Implementation of ICARS will supported structured training and assessment of NTS within the robotic surgical curriculum.
