Validity Evidence for the Competency-Based Leadership Assessment Resource (CLAR) Tool in Assessing Pediatric Fellows

Our goal was to obtain validity evidence (response process, internal structure, and relationship to other variables) for a novel Competency-based Leadership Assessment Resource (CLAR) tool for providing formative assessments of pediatric subspecialty fellows.

Faculty from 49 U.S. pediatric fellowships completed assessments over 2 cycles (2021-2022). Faculty and program directors provided feedback on content, value, and ease of use. Internal structure was assessed through Cronbach's alpha. Generalizability theory analysis explored sources of variance. Correlations were measured between the CLAR score and ACGME leadership Milestones levels, and supervision levels for the "Leading an Interprofessional Healthcare Team" Entrustable Professional Activity (EPA). Relationships between CLAR score and other variables were examined using linear mixed effect regression models.

288 faculty provided 1,035 assessments on 266 fellows over 2 cycles. The median number of assessments per fellow in each cycle was 3 (IQR 3-3 Cycle 1, IQR 2-3 Cycle 2). Fellowship training year was evenly distributed (34% 1^st, 35% 2^nd, 30% 3^rd-5^th). Critical care, child abuse, and hospital medicine were the most common subspecialties. Most program directors rated the tool as valuable. Faculty found the tool easy to use. Internal consistency was high (alpha 0.97). The greatest sources of variance were the fellow (34%), faculty in a program (17%), and subspecialty (11%). Correlation between the CLAR score and the EPA supervision levels and leadership-related Milestones were positive. Scores also increased with time in training.

The validity evidence for the CLAR tool supports use in formative assessment of pediatric fellows.