The Predictive Power of Short Answer Questions in Undergraduate Medical Education Progress Difficulty

To better understand the link between formative assessments and progress difficulty, we conducted an analysis in the undergraduate MD program of the Michael G. DeGroote School of Medicine by comparing formative assessment scores on Short Answer Questions (SAQ) called Concept Application Exercises (CAE) with subsequent progress difficulty. CAE scores are designed to formatively assess knowledge translation. These scores are not formally incorporated into the progress decision at the end of each curricular unit, which is holistic in nature. Students are referred to a student progress remediation committee if they fail to meet the curricular objectives. We sought to investigate the following research question: Do short answer questions, in the form of CAEs, predict subsequent learner progress difficulty?

Data from the last four student cohorts of 2022-2025 were included. To address the predictive power of CAE score characteristics, a binary logistic regression model was constructed with remediation committee referral as the dependent variable and CAE score characteristics as the independent variable.

This study found that the average CAE score is the most powerful predictor of later progress difficulty, with each point drop in average score associated with a 37% increase in the odds of referral to the remediation committee.

These findings illustrate the predictive value of the SAQ to identify later progress difficulty.