A comparison of methods for dealing with troublesome examination questions

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Identifying problematic examination questions is crucial for improving assessment quality.
• Traditional post-exam challenge committees can be time-consuming and retrospective.

Purpose of the Study:

• To compare the effectiveness of a student-led "question citation" procedure during exams versus a post-exam "challenge committee" process.
• To determine if the citation procedure can accurately identify troublesome questions.

Main Methods:

• Compared challenged questions (post-exam) and cited questions (during-exam) from eight examinations for second-year medical students.
• Analyzed percentage of correct responses (%CR) and discrimination index (DI) for both question types.
• Correlated the number of citations with %CR and DI.

Main Results:

• Challenged questions had significantly lower %CR than unchallenged questions.
• No significant difference in DI was found between challenged and unchallenged questions.
• A higher number of citations for a question correlated with a lower %CR, indicating troublesome questions.

Conclusions:

• The student "question citation" procedure during exams is a viable and simpler alternative to post-exam challenge committees.
• This method effectively identifies questions that may require review or revision.
• Implementing a citation procedure can streamline the process of improving examination quality.