Rasch-Based Calibration of Step-Based Task Achievement in a Surgical Clerkship: A Pilot Study of the Step Ladder...

Background Competency-based education requires intended competency levels to be aligned with learners' observed performance in authentic clinical settings. The Step Ladder System (SLS) is a three-step framework designed to structure clinical task progression and supervision expectations during clerkships. However, the empirical alignment between predefined SLS step categories and observed task-achievement patterns has not been examined. Aim This study aimed to use Rasch calibration to examine the empirical alignment between predefined SLS step categories and observed task-achievement patterns in a pilot surgical clerkship dataset. Methods We conducted a secondary analysis of physician-confirmed binary task-achievement records from 24 sixth-year medical students during a four-week gastrointestinal and pediatric surgery clerkship using a 159-task SLS checklist. Task achievement was coded as 1 if achieved at least once and 0 if not achieved. Rasch-calibrated item difficulty (β), interpreted as observed achievement difficulty within the clerkship context, was estimated using a dichotomous Rasch model with conditional maximum likelihood estimation (eRm package (R version 4.5.2, R Foundation for Statistical Computing, Vienna, Austria)). Items with invariant response patterns, defined as all-0 or all-1 responses, were summarized descriptively by step and response pattern and excluded from β estimation. Stepwise differences in β were examined using Kruskal-Wallis tests with Bonferroni-adjusted Dunn post hoc comparisons. The association between β and achievement rate was assessed using Spearman's ρ as a parameter-orientation check. Results Of the 159 SLS tasks analyzed (Step 1: 59; Step 2: 53; Step 3: 47), 36 showed invariant response patterns and were excluded from β estimation (Step 1: 2; Step 2: 14; Step 3: 20). Among these, 35 were all-0 items and one was an all-1 item; all invariant items in Steps 2 and 3 were all-0. β-based analyses included 122 basic/free item parameters under the sum-to-zero parameterization (Step 1: 56; Step 2: 39; Step 3: 27). Median β values (IQR) increased across steps: -0.55 (-1.06 to 0.46) for Step 1, 0.46 (0.07 to 1.76) for Step 2, and 0.97 (0.46 to 0.97) for Step 3. β differed significantly across steps (χ²(2) = 27.48, p < 0.001, ε² = 0.214). Post hoc comparisons showed that Step 1 had significantly lower observed achievement difficulty than Steps 2 and 3 (both p < 0.001), whereas Steps 2 and 3 did not differ significantly (p = 0.712). β was strongly and inversely associated with achievement rate (ρ = -0.992, p < 0.001), consistent with the expected orientation of the item-difficulty estimates. Conclusions Rasch calibration of SLS achievement-log data suggested that Step 1 tasks were empirically distinct from higher-step tasks in this pilot surgical clerkship dataset, whereas Step 2 and Step 3 tasks were not clearly separated among the analyzed items, raising an unresolved question of whether this reflects pilot- scale measurement limitations or structural overlap in upper-step task definitions. The concentration of all 0 invariant items in higher steps suggests that limited task exposure and sparse achievement patterns may have affected upper-step calibration. These findings support the feasibility of using Rasch-calibrated achievement logs as a curriculum diagnostic tool to identify areas where predefined step categories, task definitions, and clerkship opportunities may require refinement.