The Effect of Option Homogeneity in Multiple-Choice Items

Area of Science:

• Educational Measurement
• Psychometrics
• Natural Language Processing

Background:

• Previous research indicates option homogeneity in multiple-choice items influences item difficulty.
• Established item development guidelines often consider option homogeneity.

Purpose of the Study:

• To empirically test the effect of option homogeneity on item difficulty, discrimination, model fit, and response time.
• To assess the predictability and magnitude of changes due to option homogeneity in a professional licensure examination.

Main Methods:

• Subject matter experts and a natural language processing algorithm were used to determine option similarity to the key.
• Data from a professional licensure examination were analyzed to evaluate psychometric properties.

Main Results:

• No consistent effect of option homogeneity was found on item difficulty, discrimination, fit to the measurement model, or response time.
• Results contradicted previous findings suggesting an impact of option homogeneity.

Conclusions:

• The study's negative findings challenge current item development guidelines regarding option homogeneity.
• A hypothesis is proposed to explain the discrepancy between this study and prior research on option homogeneity effects.