The don't know option in progress testing

Area of Science:

• Medical Education
• Psychometrics
• Assessment

Background:

• Formula scoring (FS), which includes a don't know option (DKO) and penalizes incorrect answers, is debated for its impact on the validity and reliability of educational assessments.
• The DKO choice can be influenced by factors beyond knowledge, such as risk-taking tendencies, potentially introducing construct-irrelevant variance.

Purpose of the Study:

• To evaluate the effect of FS on the construct validity and reliability of radiology resident progress test scores.
• To determine if FS introduces construct-irrelevant variance into knowledge measurement.

Main Methods:

• A radiology resident progress test was administered in parallel forms (A and B) with both FS and number-right (NR) scoring versions.
• 337 participants were randomly assigned to receive either A-FS followed by B-NR or B-FS followed by A-NR.
• Hierarchical regression analysis and Cronbach's alpha were used to assess construct validity and score reliability.

Main Results:

• FS introduced construct-irrelevant variance, as DKO and its interaction with gender explained 8% of the variance in FS scores.
• Knowledge level (NR score) explained a significant portion (27%) of the variance in FS scores.
• FS scores demonstrated considerably higher reliability (mean Cronbach's alpha 0.74) compared to NR scores (mean Cronbach's alpha 0.62).

Conclusions:

• Formula scoring may impair construct validity by introducing irrelevant variance related to test-taking strategies.
• Despite potential validity concerns, FS enhances the reliability of progress test scores.
• Decisions regarding the use of FS in progress testing require a careful balance between systematic and random measurement error.