Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study...

Area of Science:

• Medical Education
• Assessment and Evaluation
• Clinical Competency

Background:

• Standardized patient encounters are crucial for assessing clinical skills.
• Ensuring consistent scoring across multiple institutions presents a significant challenge.
• Developing robust validity evidence for local graduation competency examinations is essential.

Purpose of the Study:

• To evaluate the validity of scores from a local graduation competency examination administered across seven medical schools.
• To establish standardized rater training protocols and scoring guidelines for patient notes (PNs).
• To provide a framework for reliable and valid assessment of clinical skills.

Main Methods:

• A multisite study involving 990 students across seven medical schools.
• Development and sharing of standardized clinical cases.
• Implementation of rater calibration and collaborative development of scoring guidelines.
• Analysis of scores using descriptive statistics and generalizability studies (G-studies).

Main Results:

• Validity evidence was confirmed across multiple domains, including response process, internal structure, and consequences.
• Patient notes (PNs) showed justification of differential diagnosis as the most discriminating task.
• Generalizability studies indicated a minimal school effect (<1% variance), but identified differences in PN scores across cases and schools.
• Optimal composite score reliability was achieved when PNs were weighted between 30% and 40%.

Conclusions:

• Scoring rubrics and case-specific guidelines provide rigor and feedback for medical learners.
• Variability in patient note scores across sites may reflect diverse approaches to teaching clinical reasoning.
• The study supports the use of shared cases and standardized scoring for valid competency assessments.