Validity evidence for the new ENDO Mentor Suite and its use in the Fundamentals of Endoscopic Surgery examination

The Fundamentals of Endoscopic Surgery (FES) manual skills exam is a simulation-based assessment originally developed on the GI Mentor II (GIM) with strong validity evidence. Surgical Science has since released the ENDO Mentor Suite (ES), for which validity evidence was previously demonstrated in a small, randomized study. However, there were minor scoring discrepancies for Loop Reduction and Mucosal Inspection. This follow-up study aims to determine if the ES introduces a threat to validity in the form of construct irrelevant variance through analysis of FES examinee performance.

Retrospective analysis of FES examinee data, including demographics and performance, was conducted following the integration of the ES as a testing platform in 2023. Task and total scores were obtained through a standardized computer-based algorithm. Examinees without self-reported demographics were excluded from analysis. Performance and pass rates were compared between ES and GIM examinees using standard statistical methods and sensitivity analyses within a non-inferiority framework.

Of 1,121 FES examinees, 436 (39%) provided demographic information. Simulator groups were similar in demographics and endoscopy experience. There were small statistically discernable differences in Retroflexion and Tool Targeting scores between ES and GIM examinees. However, total scores (ES 70 ± 12 vs GIM 69 ± 11, p = 0.55) and pass rates (ES 88% vs GIM 83%, p = 0.25) were similar. Sensitivity analyses also confirmed a similar pattern of non-inferiority.

The ES does not introduce construct irrelevant variance, supporting its use in the FES manual skills exam. This is demonstrated by similar overall performance and pass rates between ES and GIM examinees. Additionally, previously reported variability in Loop Reduction and Mucosal Inspection were not redemonstrated within this larger sample. Given that over half of examinees were excluded from this analysis, measures to improve quality control on demographic reporting will be essential for additional validity investigations.