Comparing Apples to Apples: Validity Evidence for an Innovative Practical Trials Approach to Evaluating the...

Area of Science:

• Medical Education Research
• Curriculum Development and Evaluation
• Quantitative Research Methods

Background:

• Evaluating medical education curricula is complex, with traditional methods often lacking validity.
• Pre-post study designs can lead to flawed interpretations and decision-making in curriculum assessment.
• This study introduces an innovative approach using practical trials and non-inferiority designs for robust curricular evaluations.

Purpose of the Study:

• To assess the impact of a significant reduction in in-person anatomy lab time on medical students' exam performance.
• To compare student outcomes between cohorts before and after a curriculum modification.
• To explore how accounting for exam difficulty and student ability influences the interpretation of curricular changes.

Main Methods:

• A non-inferiority study design was employed, comparing two large medical student cohorts (n1=304, n2=294).
• Exam performance data were analyzed, considering adjustments for faculty-determined exam difficulty (Angoff scores) and student ability (MCAT, GPA).
• The 99% confidence interval and an a priori non-inferiority margin of 10% were used to determine non-inferiority.

Main Results:

• Raw exam scores showed no significant difference (0.13%) between cohorts.
• After adjusting for exam difficulty, the new curriculum group scored 4.97% lower (p < 0.001).
• Further adjustment for student ability reduced the difference to 4.27% (p < 0.001); however, non-inferiority was established.

Conclusions:

• The study demonstrates a novel, valid approach to evaluating curricular changes in medical education.
• Reduced anatomy lab time was found to be non-inferior in terms of student exam performance.
• The findings highlight the importance of accounting for cohort differences and employing appropriate statistical designs for accurate curricular assessment.