An Analysis of Written and Numeric Scores in End-of-Rotation Forms from Three Residency Programs

Area of Science:

• Graduate Medical Education
• Medical Education Assessment
• Healthcare Quality Improvement

Background:

• End-of-Rotation Forms (EORFs) are crucial for evaluating resident performance in graduate medical education.
• Existing literature suggests that both numerical grades and written comments on EORFs tend to be overly positive.
• The utility of EORFs for identifying resident deficiencies and informing Clinical Competency Committee (CCC) discussions is questionable.

Purpose of the Study:

• To assess the informational value of EORFs for residents, program directors, and CCCs.
• To determine if EORFs from multiple programs, specialties, and institutions provide actionable feedback.
• To analyze the relevance and orientation of written comments within EORFs.

Main Methods:

• Collected 4869 EORFs from three graduate medical education programs (Internal Medicine and Anesthesia) across two institutions.
• Two researchers independently coded written comments for relevance (specificity, actionability) and orientation (praise, critical) using a standardized rubric.
• Analyzed numeric scores using descriptive statistics.

Main Results:

• Less than 1% of numeric scores indicated below-expected performance.
• Over 70% of written comments were deemed irrelevant to resident progress.
• Critically relevant comments, essential for constructive feedback, were infrequently observed across all programs.

Conclusions:

• The high rate of irrelevant comments and the scarcity of constructive feedback in EORFs are insufficient for CCCs to make informed decisions.
• Findings were consistent across programs, specialties, and institutions, indicating potential systemic issues.
• Local and systemic changes are recommended to improve the quality and utility of EORFs for resident assessment.