An investigation of the difficulty of computer-based case simulations

This study investigated the characteristics of computer-based case simulations (CCS) that may be associated with case difficulty. Difficulty was defined as the average rating by physicians of examinee performance on a nine-point scale or the passing rate on the cases. Two data sets were used, one from an administration of 18 cases, the other from an administration of 22 cases with 13 cases used on both occasions. Stepwise regression procedures were used separately for case properties and for analytic scoring of key variables to identify the best sets of predictors of case difficulty. Because of the small number of cases, regression results were evaluated for consistency across both data and both difficulty measures. For key variables, the best set of predictors included the number of different serious errors of commission, risk items, and benefit items. In general, cases were more difficult for higher values of these variables. For case variables, the only consistent variable was the length of the paragraph that provided patient history, with longer paragraphs associated with more difficult cases. Other variables were less consistent, but were often related to the structure of the simulation or the severity of the patient condition. Although the findings for case variables were limited, the analyses were very helpful in illuminating the interconnections among the variables within cases.