Implementation and Evaluation of the Z-Score System for Normalizing Residency Evaluations

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Clinical competence assessment is crucial for residency programs.
• Valid and reliable measurement tools are essential for accurate evaluations.
• The Baker Z-score system offers a method for assessing core competencies and clinical performance.

Purpose of the Study:

• To validate the use of summative scores from the Baker Z-score system.
• To estimate the number of evaluations required for reliable clinical competence measures.
• To assess the generalizability of the Baker Z-score system in residency training.

Main Methods:

• Generalizability and decision studies were conducted to estimate variance components.
• ANOVA was used to determine variance components for raw and Z-transformed scores.
• Evaluations were selected to represent residents frequently assessed by faculty.

Main Results:

• Principal component analysis showed a single factor explaining significant variance in scores.
• The major variance component was resident interaction with measurement occasions.
• 30-58 evaluations were needed for 90% reliable measurements with raw scores, and 55-57 for Z-transformed scores.

Conclusions:

• The Baker Z-score system demonstrated moderate reliability in assessing clinical competence.
• Raw absolute scores required fewer assessment occasions for reliable measurement.
• The findings suggest potential generalizability of the Baker system to other training programs.