A Prelude to Modeling the Expert: A Generalizability Study of Expert Ratings of Performance on Computerized Clinical...

Context: The University of Iowa College of Medicine has developed a series of computer-based clinical simulations and successfully integrated them into the clinical clerkship curriculum. The computerized patient simulations provide a high degree of realism in simulating a clinical encounter. In an effort to improve the validity of our clinical skills assessment, we have initiated testing research utilizing these simulations. Because of the high costs associated with employing expert raters for performance scoring, automated scoring was deemed essential.Purpose: This study is designed to address the preliminary research questions related to utilizing the simulations for performance assessment and developing a psychometrically sound automated scoring mechanism. Specifically, it addresses issues of reliability in relation to rater and simulation characteristics, and provides essential data required for designing a sound methodology to obtain ratings for modeling.Design: The judgements of 3 expert clinician/raters, grading the responses of 69 third-year medical students, to 2 computerized simulations, are analyzed in a generalizabilty study. A random effects (persons by raters) ANOVA was performed to estimate variance components for modeling. A case facet was added to the anlaysis to provide data regarding performance assessment characteristics. Estimation of the magnitude of each variance component represents the outcome of the generalizability study. Variance estimates are used in the decision study phase of the research.Results: Only moderate levels of inter-rater reliability were obtained. Four or more raters were indicated to obtain adequate reliability. A high level of task/simulation specificity was found. Three or more simulations were indicated for performance assessment. Suggestions for improving ratings were offered.