Exploratory analysis on the usage of Pi-score algorithm over endoscopic stone treatment step 1 protocol

Area of Science:

• Medical Education
• Surgical Training
• Urology

Background:

• The Performance Improvement score (Pi-score) is a validated tool for assessing performance enhancement in hands-on training.
• Endoscopic Stone Treatment (EST) step 1 is a widely adopted protocol for practical training in urology.

Purpose of the Study:

• To adapt and evaluate the reliability of the Pi-score for the Endoscopic Stone Treatment step 1 (EST s1) protocol.
• To determine if the Pi-score algorithm can objectively quantify performance improvement in EST s1 training.

Main Methods:

• The Pi-score algorithm was applied to data from 'ART in Flexible Course' training sessions in Barcelona and Milan.
• Time measurements and error counts from the first and fifth repetitions of training tasks were analyzed.
• Expert tutors' scores were compared with the algorithm's Pi-scores using Kappa statistics for inter-rater reliability and correlation analysis.

Main Results:

• Inter-rater reliability among 16 expert tutors showed fair to slight concordance across four tasks (0.10-0.30).
• The Pi-score algorithm demonstrated substantial to moderate concordance with average expert scores (0.46-0.74).
• The algorithm provided objective scores comparable to expert assessments for performance improvement.

Conclusions:

• The Pi-score is effectively adaptable for the Endoscopic Stone Treatment step 1 (EST s1) training protocol.
• The algorithm offers an objective measure of performance improvement, correlating well with expert evaluations after limited training.
• This validates the Pi-score as a reliable tool for assessing skill acquisition in EST s1.