Distractors with information in multiple choice items: a rationale based on the Rasch model

Area of Science:

• Psychometrics
• Educational Measurement
• Item Response Theory

Background:

• Existing research explores using distractor information in multiple-choice items to gauge respondent proficiency.
• The Rasch model's implications for partial credit in distractors are often overlooked, leading to pooled response probabilities.

Purpose of the Study:

• To develop and test a hypothesis for identifying informative distractors within multiple-choice items.
• To demonstrate how the partial credit parameterization of the polytomous Rasch model can be applied to distractor analysis.

Main Methods:

• Formulating a hypothesis by analyzing the shape of distractor response curves across a proficiency continuum.
• Testing the hypothesis by scoring responses (correct=2, hypothesized distractor=1, other distractors=0) and applying the polytomous Rasch model.
• Utilizing evidence such as response fit at thresholds and the order of threshold estimates to validate distractor information.

Main Results:

• The study provides a framework for forming and testing hypotheses about distractor information.
• Evidence from fit statistics and threshold estimates can determine if a distractor contains meaningful information about proficiency.
• An illustrative example demonstrates the practical application of the proposed methodology.

Conclusions:

• Informative distractors can be identified and analyzed using the polytomous Rasch model, challenging the pooling of all incorrect responses.
• This approach allows for a more nuanced understanding of respondent knowledge and the quality of assessment items.
• The methodology offers a statistically sound way to leverage distractor data for improved measurement.