The construct and criterion validity of the multi-source feedback process to assess physician performance: a...

Area of Science:

• Medical Education
• Health Professions Education
• Performance Assessment

Background:

• Physician and surgeon performance evaluation is critical for patient safety and quality of care.
• Multi-source feedback (MSF) is increasingly used for assessing healthcare professionals.
• Robust validity evidence is essential for the reliable implementation of MSF.

Purpose of the Study:

• To conduct a meta-analysis on the construct and criterion validity of multi-source feedback (MSF).
• To evaluate the effectiveness of MSF in assessing physicians and surgeons in practice.

Main Methods:

• Systematic literature search of multiple databases (PubMed, MEDLINE, CINAHL, EMBASE, PsycINFO) from 1975 to 2012.
• Inclusion of 35 studies reporting on physician/surgeon performance measures.
• Independent coding of articles and calculation of effect sizes (d) using a random-effects model.

Main Results:

• Construct validity coefficients for MSF ranged from d=0.14 to d=1.78 across different practice levels and occasions.
• Concurrent validity coefficients ranged from d=0.50 to d=0.57 based on assessor group ratings.
• Predictive validity coefficients ranged from d=1.28 to d=1.43 for MSF on standardized performance measures.

Conclusions:

• The construct and criterion validity of the MSF system is supported by the meta-analysis.
• MSF demonstrates small to large effect size differences in assessing physician/surgeon performance.
• Findings support the use of MSF across clinical and nonclinical domains.