Interrater agreement and predictive validity of faculty ratings of pediatric residents

In the study reported here attending faculty members completed a 14-item rating scale at the end of each clinical rotation in a general pediatrics residency training program. The rating scale items assessed medical knowledge, interpersonal and professional relationships, and attitudes toward education. Significant interrater agreement occurred primarily on those items concerning medical knowledge, and ratings on these items showed significant correlation with the external criterion of scores on the intraining examination of the American Board of Pediatrics. There was also significant interrater agreement for ratings on the item regarding relationship with coworkers. Intercorrelations of ratings on this item with those concerning medical knowledge were relatively low. There was no correlation between scores on the relationship item and subsequent intraining examination scores.