Cracking the code: residents' interpretations of written assessment comments.

Area of Science:

• Medical Education
• Qualitative Assessment
• Professional Development

Background:

• Growing interest in using qualitative data for resident assessment.
• Written comments on in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty.
• Potential loss of educational opportunity if residents misinterpret assessment comments.

Purpose of the Study:

• To explore residents' interpretations of written assessment comments.
• Utilize mixed methods to understand resident perspectives on ITERs.
• Investigate how residents make sense of narrative feedback.

Main Methods:

• Twelve internal medicine postgraduate year 2 (PGY2) residents ranked anonymized postgraduate year 1 (PGY1) residents based on ITER comments.
• Generalisability theory assessed inter-rater reliability.
• Constructivist grounded theory analyzed interviews on residents' interpretation processes and views on ITERs.

Main Results:

• Resident rankings of peers based on comments showed high correlation with faculty rankings (r = 0.90).
• Residents demonstrated ability to interpret 'hidden codes' and nuances in narrative comments, similar to faculty.
• Residents identified challenges with vague language and emphasized the value of face-to-face discussions and aggregated comments.

Conclusions:

• Residents effectively interpret the 'hidden code' within assessment language.
• Resident ability to rank peers using comments aligns with faculty assessment capabilities.
• Findings support the utility of narrative comments and subjective interpretation in resident assessment.