Exploring how complex multiple-choice questions could contribute to inequity in introductory physics

High-stakes exams significantly impact introductory physics students' final grades and have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Certain types of exam questions may contribute more than other types to the observed equity gaps.

The primary objective of this study was to determine whether complex multiple-choice (CMC) questions may be a potential cause of inequity.

We used four years of data from Problem Roulette, an online, not-for-credit exam preparation program, to address our objective. This data set included 951 Physics II (Electricity and Magnetism) questions, each of which we categorized as CMC or non-CMC. We then compared student performance on each question type and created a multi-level logistic regression model to control individual student and question differences.

Students performed 7.9 percentage points worse on CMC questions than they did on non-CMC questions. We find minimal additional performance differences based on student performance in the course. The results from mixed-effects models suggest that CMC questions may be contributing to the observed equity gaps, especially for male and female students, though more evidence is needed.

We found CMC questions are more difficult for everyone. Future research should examine the source of this difficulty and whether that source is functionally related to learning and assessment. Our data does not support using CMC questions instead of non-CMC questions as a way to differentiate top-performing students from everyone else.