Evaluating the utility, reliability, and validity of a resident performance evaluation instrument

Area of Science:

• Medical Education
• Psychiatry Training
• Assessment Tools

Background:

• Assessing resident performance is crucial in child and adolescent psychiatry (CAP) training.
• Existing evaluation methods may not fully capture the nuances of resident skill development.
• Standardized assessment instruments are needed to ensure consistent and reliable evaluation.

Purpose of the Study:

• To develop and validate the Global Assessment Evaluation (GAE) instrument for assessing CAP resident performance.
• To ensure the GAE is useful, valid, and reliable for evaluating critical skills.
• To provide a model for other training programs to create their own assessment tools.

Main Methods:

• The GAE was developed in three stages, informed by ACGME core competencies.
• Faculty used the instrument to evaluate residents during and after rotations.
• Statistical analyses, including intra-class correlation and Principal Components Analysis, were used for validation and reliability testing.

Main Results:

• The GAE demonstrated overall and construct validity, differentiating resident performance levels.
• High reliability was confirmed with an intra-class correlation coefficient of 0.945 for the overall score.
• Resident performance did not correlate with rotation site or model of care, indicating the GAE's focus on core competencies.

Conclusions:

• The GAE is a valuable tool for monitoring resident progress, strengths, and weaknesses in CAP training.
• Longitudinal data from the GAE provide a comprehensive view of performance, aligning with ACGME requirements.
• The GAE supports continuous improvement in resident education and evaluation practices.