Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions

Area of Science:

• Medical Education
• Assessment and Evaluation
• Psychometrics

Background:

• Effective multiple-choice questions (MCQs) are crucial for medical education assessment.
• Distractor efficiency (DE) is a key metric for evaluating MCQ quality.
• Low to medium DE items often indicate underlying item writing flaws.

Purpose of the Study:

• To identify and analyze item writing flaws in low to medium distractor efficiency (DE) multiple-choice questions (MCQs).
• To understand how specific flaws impact the effectiveness of distractors in MCQs.

Main Methods:

• Qualitative document analysis of MCQs and their item analysis reports.
• Categorization of distractor efficiency (DE) as low, medium, or high based on non-functional distractors (NFDs).
• Identification and coding of item flaws into categories: within-option, alignment, and other flaws.

Main Results:

• A significant proportion of items exhibited low (20%) or medium (42%) distractor efficiency.
• Common flaws included irrelevant distractors (16%), logic cues (12.5%), and linguistic cues (10%).
• Low cognitive level (40%) and complicated stems (11.6%) were also prevalent issues unrelated to distractors.

Conclusions:

• Item writing flaws, particularly those affecting distractors and stem clarity, negatively impact distractor efficiency.
• Addressing these identified flaws can improve the quality and effectiveness of MCQs in assessments.
• Analysis of low to medium DE items offers valuable insights for enhancing item development.