From Assistance to Autonomy: Evaluating Procedural Competency in Pediatric Emergency Medicine

Area of Science:

• Medical Education
• Pediatric Emergency Medicine
• Procedural Competency Assessment

Background:

• Pediatric emergency medicine (PEM) physicians need extensive procedural skills for pediatric emergencies.
• Fellows require supervised practice and feedback to achieve procedural competence.
• A validated tool is needed to assess PEM fellows' procedural skills.

Purpose of the Study:

• To develop and validate an evaluation tool for assessing procedural competency in PEM fellows.
• To gather evidence supporting the tool's validity in measuring skill acquisition.

Main Methods:

• Retrospective study of PEM fellows' procedural performance using a five-point entrustment/supervision scale.
• Evaluated three common procedures: laceration repair, intubation, and procedural sedation.
• Used mixed-effects binary logistic regression to analyze changes in supervision scores by training year and assessed inter- and intra-rater reliability.

Main Results:

• Second-year fellows (aOR=4.95) and third-year fellows (aOR=10.15) showed significantly higher entrustment scores than first-year fellows.
• Strong intra-rater reliability was observed overall (ρ=0.84) and for individual procedures (intubation ρ=0.78, laceration repair ρ=0.92, sedation ρ=0.85).
• Inter-rater reliability was poor; however, first-year fellows showed specialty differences in intubation scores, which diminished by the third year.

Conclusions:

• A supervision-based evaluation tool effectively captures increasing procedural competency in PEM fellows.
• The tool demonstrates strong intra-rater reliability and supports the assessment of skill progression.
• This evaluation method can facilitate standardized assessment and provide meaningful feedback during fellowship training.