Related Experiment Video
Updated: Jan 4, 2026
Using Learning Outcome Measures to assess Doctoral Nursing Education
Published on: June 21, 2010
Can Nonclinician Raters Be Trained to Assess Clinical Reasoning in Postencounter Patient Notes?
Rachel Yudkowsky1, Abbas Hyderi, Jaime Holden
1R. Yudkowsky is professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-2145-7582. A. Hyderi is professor, Department of Clinical Science, and founding senior associate dean for medical education, Kaiser Permanente School of Medicine, Pasadena, California; ORCID: https://orcid.org/0000-0002-8521-7510. J. Holden is research assistant, Department of Medical Education, University of Illinois at Chicago College of Medicine, and PharmD candidate, University of Illinois at Chicago College of Pharmacy, Chicago, Illinois. R. Kiser is associate director, Dr. Allan L. and Mary L. Graham Clinical Performance Center of the Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois. R. Stringham is associate professor of clinical medicine, Department of Family Medicine, and assistant dean for curriculum, University of Illinois at Chicago College of Medicine, Chicago, Illinois. A. Gangopadhyaya is assistant professor, Division of General Internal Medicine, Department of Medicine, associate clerkship director, M3 and M4 internal medicine, and associate course director, Doctoring and Clinical Skills, University of Illinois at Chicago College of Medicine, Chicago, Illinois. A. Khan is associate professor of clinical medicine, Division of General Internal Medicine, Department of Medicine, clerkship director, M3 and M4 internal medicine, and course director, Doctoring and Clinical Skills, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Y.S. Park is associate professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois; ORCID: http://orcid.org/0000-0001-8583-4335.
Purpose:
Clinical reasoning is often assessed through patient notes (PNs) following standardized patient (SP) encounters. While nonclinicians can score PNs using analytic tools such as checklists, these do not sufficiently encompass the holistic judgments of clinician faculty. To better model faculty judgments, the authors developed checklists with faculty-specified scoring formulas embedded in spreadsheets and studied the resulting interrater reliability (IRR) of nonclinician raters (SPs and medics) and student pass/fail status.
Method:
In Study 1, nonclinician and faculty raters rescored PNs of 55 third-year medical students across 5 cases of the 2017 Graduation Competency Examination (GCE) to determine IRR. In Study 2, nonclinician raters scored all notes of the 5-case 2018 GCE (178 students). Faculty rescored all notes of failing students and could modify formula-derived scores if faculty felt appropriate. Faculty also rescored and corrected scores of additional notes for a total of 90 notes (3 cases, including failing notes).
Results:
Mean overall percent exact agreement between nonclinician and faculty ratings was 87% (weighted kappa, 0.86) and 83% (weighted kappa, 0.88) for Study 1 and Study 2, respectively. SP and medic IRRs did not differ significantly. Four students failed the note section in 2018; 3 passed after faculty corrections. Few corrections were made to nonfailing students' notes.
Conclusions:
Nonclinician PN raters using checklists and scoring rules may provide a feasible alternative to faculty raters for low-stakes assessments and for the bulk of well-performing students. Faculty effort can be targeted strategically at rescoring notes of low-performing students and providing more detailed feedback.
Related Concept Videos
01:13Patient-centered Care
01:25Critical Thinking II
01:24Critical Thinking I
01:21Methods of Documentation III: PIE
01:24Nursing Process for Patient and Caregiver Teaching I: Assessment and Diagnosis
It is critical to determine the patient's learning needs during the assessment. Determination of learning needs compounds data...
01:25Role of Communication in the Nursing Process I: Assessment and Diagnosis
The nursing process considers the patient's emotional and physical well-being. The process can be repeated or stopped at any point if judged essential. Assessment is the first step in the nursing...