Beyond the checklist: development and validation of a multidimensional clinical competency assessment tool for...

Medical residents perform bedside procedures that affect patient outcomes. However, no national standard exists to determine when residents can perform procedures without direct supervision. The utility and feasibility of various procedural competency assessment methods remain unclear.

A multidimensional clinical competency assessment tool (CAT) was developed to evaluate procedural performance using a checklist, global skills assessment (GSA), and Entrustment Score (ES). Residents had to meet cutoffs for all 3 components to pass. Checklist cutoffs were set using the Angoff method. The CAT assessed internal medicine interns performing large-volume paracentesis (LVP) and central venous catheter insertion (CVC) across 3 hospital systems from July 2022 to June 2023. Inter-rater reliability was assessed using 2 simultaneous assessments.

The CAT was completed in 29.7% (174) of LVP and CVC procedures performed by postgraduate year-1s. The checklist passing cutoffs were <9 mistakes for LVP and <6 for CVC. Among 23 LVP dual assessments, 3 (13%) achieved passing scores with no disagreement (Cohen's kappa = 1.0). Although 85% of LVPs met the checklist cutoff, only 20% met the GSA cutoff and 15% for ES. Of 14 CVC dual assessments, none achieved passing agreement and 2 had disagreement, a percent agreement of 87%. Seventy-five percent of CVCs met the checklist cutoff, 14% of GSA, and 7% of ES.

This study demonstrates the feasibility and reliability of implementing a multidimensional clinical CAT. Checklists alone did not accurately assess competency; however, combined with holistic scores, they yielded strong overall inter-rater agreement and more appropriate pass rates for this cohort. These findings support using multiple complementary assessment methods to better identify competent performances. Future steps include refining checklists to focus on critical steps, evaluating bias in holistic scoring, and assessing the CAT over an extended period with a cohort spanning a broader range of expertise.