Related Experiment Video
Updated: Jun 7, 2026

Using Learning Outcome Measures to assess Doctoral Nursing Education
Published on: June 21, 2010
Evaluating the Quality of Artificial Intelligence-Generated Rubrics and Grading Reliability in Student-Written
Shannon Dicken1, Nicole DeLeon1, Eric J Gutierrez1
1University of Colorado Anschutz, Skaggs School of Pharmacy and Pharmaceutical Sciences, Aurora, CO, USA.
Objective:
To investigate whether generative artificial intelligence (genAI) can improve pharmacy assessment practices, specifically, rubric development, and the grading of student-written reflection assignments.
Methods:
Sample student-written reflections from an Advanced Diabetes Management elective course were used. In phase I, ChatGPT, CoPilot, and Gemini generated grading rubrics using both basic and advanced prompts. Five faculty evaluators independently assessed the 6 blinded rubrics using a 7-item scoring tool. Rubric quality was analyzed using inter-rater reliability and descriptive statistics. In phase II, 45 deidentified student reflections were graded using the CoPilot-generated rubric (advanced prompt) by 3 genAI platforms and 2 human evaluators. Scoring consistency across graders was analyzed using analysis of variance, Tukey's post hoc tests, and intraclass correlation coefficients.
Results:
Rubric quality differed significantly across genAI platforms, with moderate inter-rater reliability among faculty evaluators. Rubric quality did not differ significantly based on prompt specificity. Significant differences were observed in student reflection scores assigned by the 5 grading entities. Gemini yielded the lowest agreement with other artificial intelligence platforms and human graders in both phases.
Conclusion:
Generative artificial intelligence (genAI) demonstrates the ability to generate rubrics and score reflection assignments; however, variability across platforms and inconsistent agreement with human graders underscore the need for careful validation before educational use.
Related Concept Videos
Reliability and Validity
Self-Evaluation Maintenance Model
Social Foundations of Self III: Self-Evaluation
Self-Evaluation: Self-Enhancement and Self-Verification
Metacognition