Pediatric resident performance. The reliability and validity of rating forms

Area of Science:

• Medical Education
• Pediatric Residency Training
• Assessment and Evaluation

Background:

• Pediatric residency programs rely on faculty evaluations for resident assessment.
• Ensuring the reliability and validity of these evaluations is crucial for program quality.
• Previous studies have highlighted variability in faculty rating consistency.

Purpose of the Study:

• To analyze the reliability and validity of standard rating forms used in a large pediatric residency program.
• To assess the correlation between faculty ratings and performance on the Pediatric In-Training Program (PITE).
• To identify factors influencing rating reliability and validity over time.

Main Methods:

• Analysis of rating data from pediatric residents participating in the Pediatric In-Training Program (PITE) between 1977 and 1981.
• Calculation of reliability metrics for individual and aggregated faculty ratings.
• Correlation analysis between faculty ratings and PITE scores for different resident levels.

Main Results:

• Individual faculty ratings exhibited very low reliability.
• Aggregated ratings across multiple faculty members achieved acceptable reliability levels.
• First-year residents' history-taking ability ratings significantly correlated with PITE scores.
• Ratings for more advanced residents showed diminished correlation with PITE scores, attributed to ceiling effects.

Conclusions:

• While individual faculty ratings lack reliability, aggregating ratings enhances assessment consistency.
• Early clinical skill assessments, like history-taking, appear more reliable and valid predictors of performance.
• Ceiling effects in advanced residents' evaluations may limit the utility of rating forms for differentiating high performers.