Related Experiment Videos
Evaluating an Iteratively Revised Assessment for a Global Pediatric Surgery Short Course Using Mixed-Effects Modeling
Erin Kim1, Phillip J Hsu2, Deborah M Rooney3
1Department of Surgery, Brigham and Women's Hospital, Boston, MA.
Objective:
To evaluate changes in learner performance after a global pediatric surgery short course, measure consistency of scores across 4 iteratively revised assessment versions, and examine item-level performance and language-related bias.
Design:
Prospective pre- and postcourse assessment study with generalized linear mixed-effects models for whole-test and domain-level analyses and many-facet Rasch modeling for item-level analyses and language bias.
Setting:
Five rural hospitals (2 districts, 3 regional) in Rwanda (January-March 2025).
Participants:
Sixty nonspecialist physicians (26 interns, 23 junior physicians, 11 senior physicians). Most preferred English (n = 54), with a minority preferring French (n = 6). Two-thirds had prior pediatric surgery exposure (n = 40).
Results:
Overall scores improved significantly from pre- to postassessment, from 71.3% to 86.5% (+15.2 percentage points; p < 0.001). Across domains, mean percentage correct increased from 73.4% to 94.0% in Diagnosis, 69.7% to 83.4% in Transfer, and 70.9% to 84.7% in Management (all p < 0.001; partial eta squared 0.47-0.88). Across the 4 assessment versions, there were no significant differences in baseline scores, postcourse scores, or pre- to postcourse change for total or domain-specific scores (p > 0.05). In mixed-effects models, language preference was associated with total scores and the Management domain (p = 0.03-0.05; partial eta squared, η²p ≥ 0.41), while training level, prior pediatric surgery exposure, hospital site, and assessment version were nonsignificant (all p > 0.05). Item-level analyses identified items with misfit and language-related differential item functioning, including bias disadvantaging French-preferring learners for several items.
Conclusion:
A single-day pediatric surgery short course was associated with significant improvements in postcourse knowledge across all domains, while iterative assessment revisions preserved score comparability across versions and sites.