Basic Laparoscopic Skills Assessment Study: Validation and Standard Setting among Canadian Urology Trainees

As urology training programs move to a competency based medical education model, iterative assessments with objective standards will be required. To develop a valid set of technical skills standards we initiated a national skills assessment study focusing initially on laparoscopic skills.

Between February 2014 and March 2016 the basic laparoscopic skill of Canadian urology trainees and attending urologists was assessed using 4 standardized tasks from the AUA (American Urological Association) BLUS (Basic Laparoscopic Urological Surgery) curriculum, including peg transfer, pattern cutting, suturing and knot tying, and vascular clip applying. All performances were video recorded and assessed using 3 methods, including time and error based scoring, expert global rating scores and C-SATS (Crowd-Sourced Assessments of Technical Skill Global Rating Scale), a novel, crowd sourced assessment platform. Different methods of standard setting were used to develop pass-fail cut points.

Six attending urologists and 99 trainees completed testing. Reported laparoscopic experience and training level correlated with performance (p <0.01). Attending urologists were significantly better than trainees (p <0.05), demonstrating construct validity evidence for the 4 AUA BLUS tasks. The C-SATS method of assessment correlated well with the traditional methods of time and error based scoring, and the global rating scale. We were able to use relative and absolute standard setting methods to define pass-fail cut points for all 4 AUA BLUS tasks.

The 4 AUA BLUS tasks demonstrated good construct validity evidence for use in assessing basic laparoscopic skill. Performance scores using the novel C-SATS platform correlated well with traditional time-consuming methods of assessment. Various standard setting methods were used to develop pass-fail cut points for educators to use when making formative and summative assessments of basic laparoscopic skill.