Sharpening the eye of the OSCE with critical action analysis

When interpreting performance scores on an objective structured clinical examination (OSCE), are all checklist items created equal? Although assigning priority through checklist item weighting is often used to emphasize the clinical importance of selected checklist items, the authors propose the use of critical action analysis as an additional method for analyzing and discriminating clinical performance in clinical skill assessment exercises. A critical action is defined as an OSCE checklist item whose performance is critical to ensure an optimal patient outcome and avoid medical error. In this study, the authors analyzed a set of clerkship OSCE performance outcome data and compared the results of critical action analysis versus traditional checklist item performance scores.

OSCE performance scores of 398 third-year clerkship students from 2003 to 2006 at the University of Virginia School of Medicine were analyzed using descriptive statistics and a logistic regression model. Through a consensus process, 10 of 25 OSCE cases were identified as containing critical actions.

Students who scored above the median correctly performed the critical actions more often than those scoring lower. However, for 9 of 10 cases, 6% to 46% of higher-scoring students failed to perform the critical action correctly.

Failure to address this skill assessment outcome is a missed opportunity to more fully understand and apply the results of such examinations to the clinical performance development of medical students. Including critical action analysis in OSCE data interpretation sharpens the eye of the OSCE and enhances its value in clinical skill assessment.