Fundamentals of Laparoscopic Surgery manual test: is videotaped performance assessment an option?

In efforts to maintain standards required to evaluate the high-stakes assessment, Society of American Gastrointestinal and Endoscopic Surgeons Fundamentals of Laparoscopic Surgery (FLS) requires all new proctors to complete the train-the-proctor workshop. As the pool of FLS proctors expands, new methods to streamline training and quality assurance programs should be considered. We propose that videotaped performances of the FLS manual tasks may be an alternative proxy to live assessment for training of new proctors, but evaluation of proctors' measures from videotaped FLS performances is required before implementation.

A 2-phased research consisted of capturing newly trained proctors' (n = 20) ratings of 3 similar FLS performances across 3 stations-live (Live), videotaped-laparoscopic only (Lap Only) view, and videotaped-dual (Dual) views, during the 2012 Society of American Gastrointestinal and Endoscopic Surgeons FLS train-the-proctor workshop. A month later, a sample of proctors (n = 9) viewed videotaped versions of live FLS performances originally observed during the workshop. Captured metrics include recognition of a predefined critical error for each task (dichotomously scored and summed) and time to complete each of the 5 tasks. Analysis of variance compared the proctors' summed ratings for similar performances across Live, Lap Only, and Dual views, whereas paired t test compared recorded times of Lap Only vs Dual views, Live vs web ratings, and proctors' recorded times across the Lap Only and Dual views.

There were neither differences in ratings across Live, Lap Only, and Dual views (p = 0.49) nor in recorded times for performances viewed across Lap Only and Dual viewing options (p = 0.29 and 0.76, respectively). Mean summed performance ratings observed live (4.6) were higher than those observed via the web (4.0), although not significant (p = 0.051). There were no differences in recorded times for identical performances across Live and web observations (p = 0.18 and 0.69, respectively), although findings were limited by sample size for some tasks.

In spite of limitations, favorable results of this preliminary study supports use of videotaped FLS performances for streamlining training mechanisms for FLS proctors, and developing best practices in standard setting and long-term evaluation of the of FLS proctors.