Is It All About the Form? Norm- vs Criterion-Referenced Ratings and Faculty Inter-Rater Reliability

Area of Science:

• Medical Education Research
• Healthcare Quality Improvement
• Performance Assessment

Background:

• Limited research exists on the data quality of resident performance evaluations.
• This study addresses the need to compare different evaluation scaling approaches.

Purpose of the Study:

• To compare inter-rater reliability between norm-referenced and criterion-referenced evaluation scaling methods.
• To assess which scaling approach yields more reliable data for faculty performance evaluations of residents.

Main Methods:

• Examined resident performance evaluation data from 426 residents across 3 programs at 2 institutions.
• Calculated faculty inter-rater reliability using intraclass correlation coefficients (ICCs) for criterion-referenced and norm-referenced forms.
• Analyzed reliability across competency areas, evaluation forms, and scaling types.

Main Results:

• Criterion-referenced scaling demonstrated higher average inter-rater reliability across all competencies compared to norm-referenced scaling.
• Aggregate scores for criterion-referenced scaling showed significantly higher reliability (z=1.37) than norm-referenced scaling (z=0.88).
• Distributions of composite scores indicated criterion-referenced evaluations better represented the resident performance continuum.

Conclusions:

• Criterion-referenced evaluation approaches provide superior inter-rater reliability compared to norm-referenced approaches.
• Using criterion-referenced scaling may lead to more valid data in resident evaluations.
• Further research is needed to establish best practices for resident evaluation.