Comparing Apples to Apples: Validity Evidence for an Innovative Practical Trials Approach to Evaluating the...

Evaluating curricular changes in medical education remains challenging due to their complexity. Traditional pre-post comparisons can lack sufficient validity evidence, which can result in flawed interpretations and decision-making. This study proposes an innovative approach using practical trials and non-inferiority designs to strengthen the validity of curricular evaluations.

We conducted a non-inferiority study comparing medical students' exam performance between two large cohorts (n₁ = 304; n₂ = 294) before and after a curricular change that significantly reduced in-person anatomy lab time. We explore how the results and conclusions may vary based on whether cohort differences in exam difficulty (using faculty-determined Angoff scores) and student ability (using MCAT and GPA data) are accounted for, and what a priori non-inferiority margin is used.

Raw exam scores showed no significant difference between cohorts (Mean difference = 0.13%). However, after adjusting for exam difficulty, students in the new curriculum scored 4.97% lower, p < 0.001. Adjusting for student ability further reduced this difference to 4.27%, p < 0.001. Using the 99% confidence interval for our mean differences and an a priori non-inferiority margin of 10%, we determined the two curricula were non-inferior.

This study demonstrates a novel approach to evaluating curricular changes. While we concluded that reduced anatomy time did not meaningfully decrease student performance, our analyses also reveal how various interpretations can arise based on the approach to comparison. We offer guidance on how practical trials can be combined with strategies to account for cohort differences to enhance the validity of curricular comparisons.