Validation of computer-mediated open-ended standardized patient assessments

Quantitative information regarding the development of students' clinical reasoning skills is valuable in assessing third- and fourth-year medical students' clinical knowledge. Standardized patient cases are often used to obtain this quantified information. These cases typically involve a post-patient examination consisting of a series of closed-ended multiple-choice questions. Many medical educators question whether the results from the closed-ended multiple-choice questions truly reflect students' clinical knowledge and reasoning proficiency. Since 1995, the Kirksville College of Osteopathic Medicine (KCOM) has used a computer program, SOAP Note Plus, for standardized patient assessment of students to automate the post-encounter documentation and evaluation. This paper describes the development of the SOAP Note Plus program and a validation study which was conducted to determine the empirical association between the computer-mediated closed-ended and open-ended post-encounter standardized patient assessments. Correlation and GENOVA statistics were used in the analysis of the performance of third-year medical students on the closed-ended and the open-ended assessments following standardized patient encounters and the relationship to their undergraduate GPA, first 2-year medical school GPA and the actual clinical rotation evaluations. The initial results show the positive relationship between the open-ended assessment and the actual clinical rotation evaluations. undergraduate GPA and the first two-years of medical school GPA.