Related Experiment Video
Updated: Sep 27, 2026

Using Learning Outcome Measures to assess Doctoral Nursing Education
Published on: June 21, 2010
Development and evaluation of a simple structured rubric for grading short answer questions
Jaekyu Shin1, Katherine Gruenberg1, Rupa Lalchandani Tuan2
1Department of Clinical Pharmacy, School of Pharmacy, USA.
Background:
Constructed-response questions (CRQs) effectively assess pharmacy students' therapeutic reasoning but are underutilized due to inconsistent and resource-intensive grading. This study describes the development and evaluation of a simple, structured rubric template designed to address these challenges.
Methods:
A 6-level rubric used in a Doctor of Pharmacy curriculum was modified into a 3-to-4-level rubric based on a single-point rubric design. It focused on defining "borderline pass" criteria and incorporated three specific therapeutic reasoning domains (making a decision, analytical skills, and content knowledge). It was implemented across three integrated organ-system courses in the 2023-2024 academic year. Perceptions of graders, item writers, and course directors were evaluated via a survey and focus groups. Interrater reliability between expert (faculty) and non-expert (residents) graders was evaluated using the intraclass correlation coefficient (ICC) across 50 randomly selected student answers for 8 CRQs.
Results:
The survey response rate was 96%. Respondents reported high clarity regarding borderline criteria (95.7%), and over 90% of inexperienced graders found the rubric easy to use. Pre-grading calibration sessions using live student answers were highly valued (>90%). Focus group themes highlighted improved grading efficiency and item writing, alongside challenges in defining borderline criteria for basic science items. The grading consistency between expert and non-expert graders was good (ICC = 0.85; range: 0.82-0.87).
Conclusion:
The simple, structured rubric template was successfully implemented, demonstrating high clarity and good grading consistency between expert and non-expert graders. The rubric may help optimize resource utilization and support the expanded use of CRQs in pharmacy education.
Related Concept Videos
Nursing Evaluation
Section...
Surveys
Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning, patient...
Social Loafing
Reliability and Validity
Social Foundations of Self III: Self-Evaluation