What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy

Area of Science:

• Medical Education
• Educational Psychology
• Assessment Science

Background:

• Medical educators utilize Bloom's taxonomy to design multiple-choice questions (MCQs) targeting specific cognitive levels.
• There is an assumption that MCQs can effectively differentiate between higher-order (analyze, apply) and lower-order (recall) learning.

Purpose of the Study:

• To evaluate the agreement among students, faculty, and assessment experts in classifying MCQs according to Bloom's revised taxonomy levels.
• To determine if educators' intended cognitive levels of MCQs are perceived similarly by learners.

Main Methods:

• Phase 1 involved 90 MCQs being classified by students, faculty, and experts using Bloom's levels.
• Phase 2 included faculty-written MCQs (higher- or lower-order) assessed by students for their perceived cognitive level.

Main Results:

• Low interrater reliability was observed within student and faculty groups, and across all stakeholder groups in Phase 1.
• While 63% of students agreed with faculty on MCQ order in Phase 2, low agreement was found between paired faculty and student ratings.
• Assessment experts demonstrated high interrater reliability in classifying MCQs.

Conclusions:

• Faculty and students often diverge in their perception of an MCQ's cognitive demand, irrespective of the intended learning level.
• This discrepancy suggests that the intended assessment of higher-order thinking through MCQs may not be consistently achieved or recognized by students.