Comparison of Peer, Near-Peer, and AI-Assisted Methods to Faculty Grading of SOAP Notes
John Pham1, Krista L Donohoe1, Dayanjan Wijesinghe1
1Virginia Commonwealth University School of Pharmacy, Richmond, VA, USA.
Objective:
The objective of this study was to determine which approach (peer, near-peer, Chat- Chat-Generative Pretrained Transformer [GPT] Plus, or an inhouse implementation of a GPT-4-based application referred to as GRADES) was closest to faculty grading of Subjective, Objective, Assessment, Plan (SOAP) notes.
Methods:
Second-year student pharmacists (n = 83) completed a practice SOAP note. Five methods were used to grade the SOAP notes: faculty, peer, near-peer, ChatGPT Plus, and GRADES. Variability in rubric scores among grading methods was analyzed using Friedman 1-way repeated measures analysis of variance with post hoc testing. Time was tracked for each grading method to evaluate efficiency.
Results:
The median scores and IQRs for each grading method were as follows: faculty (65% [56%-71%], peer (78% [62%-88%]), near-peer (77% [72%-86%]), ChatGPT Plus (87% [80%-91%]), and GRADES (66% [57%-73%]). Peer-, near-peer, and ChatGPT scores were statistically different from faculty scores. Scores from GRADES were not significantly different from faculty grading.
Conclusion:
The use of an inhouse implementation of GPT-4 (GRADES), with faculty oversight, resulted in rubric scores similar to those of a faculty grader.
Related Concept Videos
Social Loafing
Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
Surveys
Methods of Documentation III: PIE
Nursing Evaluation
Self-Evaluation: Self-Enhancement and Self-Verification


