The Effect of the Question Mark Option in Progress Testing: A Large-Scale Longitudinal Study

Area of Science:

• Medical Education
• Assessment and Evaluation
• Psychometrics

Background:

• Traditional formula scoring in medical progress tests (PT) uses a question mark option to deter guessing.
• This feature may disadvantage risk-averse students and introduce bias due to test-taking strategies.
• Dutch medical schools transitioned to computer adaptive-PT (CA-PT) using Item Response Theory, dynamically adjusting difficulty and omitting the question mark option.

Purpose of the Study:

• To evaluate the impact of the question mark option in conventional PT on subsequent performance in CA-PT.
• To explore the relationship between question mark usage in traditional PT and student ability assessment via CA-PT.
• To analyze variability in the effect of question mark usage across different student cohorts.

Main Methods:

• Retrospective analysis of data from medical students across seven faculties who completed both conventional PT and CA-PT.
• Assessment of Z-scores for total and question mark scores (unanswered questions) in conventional PT, and theta scores in CA-PT.
• Linear modeling to determine the effect of question mark score on theta, controlling for conventional PT score; cluster analysis for student subgroups.

Main Results:

• Students with similar conventional PT scores who utilized the question mark option (left more questions unanswered) generally showed better performance on CA-PT.
• This effect was observed to diminish as students progressed through their academic studies.
• Cluster analysis indicated variable effects across student subgroups, most prominent in year 4, with a reversed effect noted in year 5.

Conclusions:

• The use of the question mark option in formula scoring significantly influences student performance, demonstrating considerable variability.
• This variability suggests that formula scoring assesses factors beyond pure knowledge, such as test-taking strategies or risk aversion.
• Aligning assessment scoring methods with their intended educational goals is crucial for accurate ability measurement.