Predicting Resident Competence for Otolaryngology Key Indicator Procedures.

Area of Science:

• Surgical Education
• Otolaryngology Training
• Competency Assessment

Background:

• Competency-based surgical education necessitates robust assessment tools and clear performance benchmarks.
• Key Indicator Procedures (KIPs) are crucial for evaluating surgical exposure during otolaryngology residency.
• The precise number of KIP evaluations required to ensure operative competence remains undetermined.

Purpose of the Study:

• To utilize Bayesian mixed-effects models to establish predicted performance expectations for Key Indicator Procedures (KIPs) in otolaryngology.
• To determine the number of evaluations needed for residents to achieve a high probability of demonstrating operative competence.

Main Methods:

• Operative performance data from 842 assessments of KIPs were collected from 92 residents (PGY 2-5) using the SIMPL OR smartphone application.
• Bayesian mixed-effects models were employed to estimate the probability of residents achieving a 'practice-ready' (PR) rating based on prior PR ratings for each KIP.
• Model predictive validity was assessed using cross-validation, with an average Area Under the Receiver Operating Curve of 0.77.

Main Results:

• The predictive model demonstrated good performance with an average Area Under the Receiver Operating Curve of 0.77.
• Estimated numbers of prior 'practice-ready' (PR) ratings required for senior residents to achieve a 95% probability of a subsequent PR rating varied by KIP.
• For example, PGY4 otolaryngology residents needed an average of 10 PR ratings for mastoidectomies, while PGY5 residents required 4 PR ratings.

Conclusions:

• Predictive modeling offers a valuable method for establishing data-driven assessment benchmarks in competency-based surgical education.
• These findings can guide the development of more objective and reliable evaluation standards in otolaryngology training.