How did you guess? Or, what do multiple-choice questions measure?

Area of Science:

• Medical Education
• Cognitive Assessment

Background:

• Multiple-choice questions (MCQs) are often designed to assess higher-order thinking skills, including problem-solving.
• An underlying assumption is that MCQs requiring more complex cognitive processes should be more difficult for students.

Purpose of the Study:

• To test the hypothesis that MCQ difficulty correlates with the cognitive process required.
• To investigate the actual process students use when answering MCQs classified as problem-solving.

Main Methods:

• A 150-item MCQ examination was administered to 721 medical students across seven Australian medical schools.
• A subset of questions was analyzed through qualitative exploration of the question-answering process with final-year students.
• Inter-rater reliability for question classification was assessed among teachers and students.

Main Results:

• No significant correlation was found between question difficulty and the cognitive process classification.
• Student self-reports indicated a strong reliance on knowledge recall rather than genuine problem-solving for these MCQs.
• Classification agreement for question type between teachers and students was only slightly better than chance.

Conclusions:

• The hypothesis that MCQ difficulty reflects the cognitive process complexity is not supported by this data.
• Current MCQ design and classification may not accurately measure students' problem-solving abilities.
• Further research is needed to develop and validate assessment methods that truly capture problem-solving skills in medical education.