Group assessments of resident physicians improve reliability and decrease halo error.

Area of Science:

• Medical Education
• Resident Assessment
• Faculty Development

Background:

• Individual faculty assessments of resident competency often suffer from inconsistent standards, low reliability, and the
• halo" effect, impacting accurate evaluation.
• Traditional assessment methods may not fully capture resident performance due to subjective biases.

Purpose of the Study:

• To determine if incorporating faculty group assessments alongside individual assessments improves reliability and reduces halo effects.
• To compare the effectiveness of combined group and individual assessment methods versus individual assessments alone.

Main Methods:

• A prospective, longitudinal study conducted in outpatient continuity clinics of an internal medicine residency program.
• Comparison of faculty-on-resident and group faculty-on-resident assessment scores.
• Analysis of inter-rater reliability and inter-item correlations.

Main Results:

• Group assessments yielded significantly higher mean scores than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001).
• Combined assessments increased overall inter-rater reliability (ICC = 0.828) compared to individual assessments alone (ICC = 0.749).
• Group assessments showed lower inter-item correlations (0.49) than individual assessments (0.68), indicating reduced halo effect.

Conclusions:

• Adding group assessments to individual faculty evaluations enhances inter-rater reliability and mitigates the halo effect in resident competency assessment.
• This feasible model offers a more reliable and discriminating method for evaluating residents in graduate medical education.
• The findings support the adoption of group assessment strategies to improve the quality of resident performance evaluations.