Evaluation of a High Stakes Physician Competency Assessment: Lessons for Assessor Training, Program Accountability,...

Area of Science:

• Medical Education
• Physician Competency Assessment
• Professional Development

Background:

• Limited evidence exists on postlicensure high-stakes physician competency assessment programs.
• Understanding physician dyscompetence is crucial for ongoing professional development.

Purpose of the Study:

• Evaluate assessor inter-rater reliability in a high-stakes physician assessment.
• Determine the relationship between individual assessment components and overall physician performance.
• Identify specific competency needs and understand physician dyscompetence more broadly.

Main Methods:

• Four assessors independently reviewed 102 video-recorded physician assessments.
• Inter-rater reliability was measured using intraclass correlation coefficients.
• Analysis of variance and linear regression examined component scores in relation to overall performance.

Main Results:

• High intraclass correlation coefficients (0.756-0.876) indicated strong inter-rater reliability, especially for overall performance.
• Individual component scores were positively associated with overall physician performance.
• Poorer performing physicians exhibited greater variability in component scores, highlighting specific areas of weakness.

Conclusions:

• High-stakes assessments can be reliably conducted and effectively identify performance gaps in potentially dyscompetent physicians.
• Assessment tools can pinpoint specific competency needs, informing targeted educational interventions.
• Recognizing that physician dyscompetence often presents as specific skill deficits is vital for career-long learning.