Do item-writing flaws reduce examinations psychometric quality?

Area of Science:

• Medical Education
• Assessment Science

Background:

• Psychometric properties of multiple-choice questions (MCQ) are influenced by anatomical site and item-writing flaws (IWF).
• Understanding these influences is crucial for improving MCQ quality in assessments.

Purpose of the Study:

• To investigate the impact of anatomical sites and IWF on the psychometric characteristics of Clinical Anatomy MCQs.

Main Methods:

• 800 Clinical Anatomy MCQs from eight examinations were analyzed.
• Items were classified based on anatomical site and the presence of IWF.
• Difficulty and discrimination indices were calculated for each MCQ.

Main Results:

• 55.8% of the MCQs contained item-writing flaws.
• Anatomical site accounted for 6.2% of difficulty variance and 3.2% of discrimination variance.
• Item-writing flaws accounted for 2.8% of difficulty variance and 0.8% of discrimination variance.

Conclusions:

• The effect of anatomical site on MCQ psychometrics was greater than the effect of IWF.
• Specific IWF categories (Writing the Stem, Writing the Choices) negatively impacted item difficulty and discrimination.
• Prioritizing anatomical topic relevance over IWF during MCQ construction is recommended.