Related Experiment Video
Updated: May 15, 2026
10:26
Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
Multiple choice questions can be designed or revised to challenge learners' critical thinking.
Rochelle E Tractenberg1, Matthew M Gushta, Susan E Mulroney
1Collaborative for Research on Outcomes and -Metrics and Departments of Neurology, Biostatistics, Bioinformatics & Biomathematics, and Psychiatry, Georgetown University Medical Center, Building D, Suite 207, 4000 Reservoir Rd. NW, Washington, DC, 20057, USA, rochelle.tractenberg@gmail.com.
Advances in Health Sciences Education : Theory and Practice
|January 5, 2013
Summary
Cognitive psychology experts can effectively rate the complexity of multiple-choice (MC) questions. This allows for increasing higher-order thinking in assessments without altering item difficulty, even when experts lack specific content knowledge.
Area of Science:
- Educational Psychology
- Physiology Assessment
Background:
- Integrating higher-order thinking into multiple-choice (MC) exams is crucial but challenging.
- Content expertise may hinder the creation of complex assessment items, as experts may not easily adopt a novice perspective.
Purpose of the Study:
- To test the independence of content expertise and cognitive complexity ratings of MC items.
- To determine if non-content experts can reliably assess the cognitive complexity of MC questions.
- To explore methods for enhancing cognitive complexity in MC questions without altering item difficulty.
Main Methods:
- Three cognitive psychology experts rated the cognitive complexity of 252 physiology MC items using a six-level matrix.
- Rasch modeling was employed to estimate item difficulties.
- Statistical analysis examined the relationship between complexity ratings and difficulty estimates.
Main Results:
- Cognitive complexity ratings were statistically independent of item difficulty estimates for 88% of the MC items.
- Modifications were identified to increase item complexity by one level without impacting item difficulty.
- Non-content experts demonstrated effectiveness in rating cognitive complexity.
Conclusions:
- Cognitive complexity of MC items can be reliably rated by non-content experts.
- A structured complexity matrix, applied by trained peer groups without domain expertise, can improve targeted item complexity.
- Higher-order thinking can be integrated into MC questions without compromising item difficulty or other test characteristics.
Related Concept Videos
01:19Critical Thinking
Critical thinking involves reflective and productive thinking and the evaluation of evidence. Critical thinkers seek to understand the deeper meaning of ideas, question assumptions, and make independent decisions about what to believe or do. Scientists, for instance, are often critical thinkers. Critical thinking also requires humility about what we know and don't know and the motivation to look beyond the obvious. It is essential for effective problem-solving.
Colleges and universities are...
Colleges and universities are...
01:25Critical Thinking II
Critical thinking is a cognitive process with several attributes. The attributes of critical thinking include the following:
01:24Critical Thinking I
Critical thinking helps decision-making and allows nurses to recognize barriers to success and find solutions to possible issues. It helps to brainstorm and implement ideas to achieve goals. Critical thinking helps acknowledge and state workflow inefficiencies while improving management techniques. Nurses understand the value of critical thinking and look for fellow nurses with critical thinking skills to upgrade their professional standards. Critical thinking can advance a nurse's career with...
01:13Patient-centered Care
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
01:26Metacognition
Metacognition is a conscious process where individuals are aware of their cognitive and executive processes, such as planning before solving a problem or self-monitoring during reading. For instance, a writer may need help with composing a piece. The situation involves a writer who is working on a piece of writing, but while doing so, they realize that something is missing. They notice that their characters lack depth or details. This realization occurs because the writer is reflecting on their...
01:19Counterfactual Thinking
Counterfactual thinking is a cognitive process wherein individuals mentally reconstruct alternative versions of past events, often beginning with “what if” or “if only.” This reflective mechanism plays a significant role in shaping emotional experiences and guiding future behavior. Though typically triggered by unfavorable or unexpected outcomes, counterfactual thinking can also emerge in mundane, everyday decisions and experiences, revealing its deep entrenchment in human cognition.Types of...