A Faculty Development Program to Reduce Rater Error on Milestone-Based Assessments.

Area of Science:

• Medical Education
• Resident Assessment
• Faculty Development

Background:

• Rater errors (halo, range restriction, leniency) threaten the validity of resident assessments by faculty.
• Standardized assessment tools are crucial for reliable evaluation of resident performance.

Purpose of the Study:

• To determine if faculty development on a new Milestone-based assessment tool reduces rater errors.
• To compare rater error rates between faculty who participated in development and those who did not.

Main Methods:

• Analysis of 412 resident Milestone evaluations completed by faculty between July 2012 and June 2013.
• Selection of the two most frequently rated Milestones per competency for detailed analysis.
• Comparison of rater error metrics between faculty participants and non-participants in a development activity.

Main Results:

• Faculty who participated in development training showed significantly fewer rater errors.
• Non-participant faculty were more likely to assign identical scores across all Milestones (16%).
• Participant faculty used a wider score range and lower minimum scores, rating residents less frequently at the highest level.

Conclusions:

• Faculty education on Milestone assessment tools demonstrably reduces rater errors.
• Training faculty in assessment tools enhances the validity and reliability of resident evaluations.
• Developing a "core faculty" cadre trained in Milestone assessment is recommended to improve resident assessment quality.