Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based...

Area of Science:

• Medical Education
• Graduate Medical Education Assessment
• Competency-Based Medical Education

Background:

• Clinical Competency Committees (CCCs) are crucial for evaluating resident performance.
• Assessing Entrustable Professional Activities (EPAs) and milestones is key for resident advancement.
• Competency-based, time-variable (CBTV) advancement models are being piloted in medical education.

Purpose of the Study:

• To evaluate the response process validity evidence for CCC assessments.
• To examine assessments of EPAs and milestones for first-year pediatric residents.
• To compare assessments within a national pilot of CBTV advancement.

Main Methods:

• Compared assessments from actual CCCs and blinded
• virtual
• CCCs for 48 first-year pediatric residents.
• Analyzed data from 2 EPAs and 8 milestones across 4 residency programs (2016-2018).
• Utilized ordinal mixed-effects models to compare assessments across different resident training paths.

Main Results:

• Actual CCCs provided higher scores than virtual CCCs for milestones and one EPA.
• Residents graduating from different institutions received lower milestone ratings.
• CBTV residents showed similar or higher milestone ratings compared to non-CBTV residents from the same institution.

Conclusions:

• First-year residents from CBTV programs demonstrated comparable competence to traditional graduates.
• Evidence suggests CCC members may rely on undocumented personal knowledge of learners.
• Further research is needed to understand the nuances of CCC assessment validity.