Related Experiment Video
Updated: Jun 7, 2026
10:07
Using Learning Outcome Measures to assess Doctoral Nursing Education
Published on: June 21, 2010
Evaluating the Quality of Artificial Intelligence-Generated Rubrics and Grading Reliability in Student-Written
Shannon Dicken1, Nicole DeLeon1, Eric J Gutierrez1
1University of Colorado Anschutz, Skaggs School of Pharmacy and Pharmaceutical Sciences, Aurora, CO, USA.
American Journal of Pharmaceutical Education
|June 5, 2026
Summary
Generative artificial intelligence (genAI) can create grading rubrics and assess student reflections in pharmacy education. However, AI platform variability necessitates careful validation for reliable educational implementation.
Area of Science:
- Pharmacy education
- Artificial intelligence in education
Background:
- Pharmacy curricula increasingly utilize reflective writing assignments to assess student learning.
- Evaluating these assignments requires consistent and reliable grading rubrics.
Purpose of the Study:
- To evaluate the efficacy of generative artificial intelligence (genAI) in developing grading rubrics and grading student reflections in pharmacy education.
- To compare the performance of different genAI platforms (ChatGPT, Copilot, Gemini) in rubric generation and assignment grading.
Main Methods:
- Phase I: Three genAI platforms generated grading rubrics for student reflections. Faculty assessed rubric quality using interrater reliability.
- Phase II: Three genAI platforms and two human evaluators graded student reflections using a standardized rubric. Scoring consistency was analyzed using ANOVA and ICC.
Main Results:
- GenAI platforms produced rubrics of varying quality, with moderate interrater reliability among faculty.
- Significant differences were found in scores assigned by genAI platforms and human graders.
- Gemini showed the lowest agreement with other AI platforms and human graders.
Conclusions:
- GenAI tools can generate rubrics and grade assignments, showing potential for pharmacy education.
- Variability in genAI performance and inconsistent agreement with human graders highlight the need for rigorous validation prior to adoption.
- Further research is needed to optimize genAI use in educational assessment.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:29Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
01:30Social Foundations of Self III: Self-Evaluation
Self-evaluation is the process by which individuals assess their abilities, behaviors, and characteristics based on feedback from others. Charles H. Cooley observed that a person’s self-perception is primarily influenced by how others see and judge them. He suggested that individuals form their identities based on their interpretations of others' reactions. As a result, social interactions play a crucial role in shaping self-esteem and personal identity. These external evaluations often blend...
03:00Self-Evaluation: Self-Enhancement and Self-Verification
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
01:26Metacognition
Metacognition is a conscious process where individuals are aware of their cognitive and executive processes, such as planning before solving a problem or self-monitoring during reading. For instance, a writer may need help with composing a piece. The situation involves a writer who is working on a piece of writing, but while doing so, they realize that something is missing. They notice that their characters lack depth or details. This realization occurs because the writer is reflecting on their...