Validation of computer-mediated open-ended standardized patient assessments

Area of Science:

• Medical Education
• Clinical Reasoning Assessment
• Health Professions Education

Background:

• Quantitative assessment of clinical reasoning is crucial for medical students.
• Standardized patient cases with multiple-choice questions are common but their validity is debated.
• The Kirksville College of Osteopathic Medicine (KCOM) developed SOAP Note Plus for automated assessment.

Purpose of the Study:

• To describe the development of the SOAP Note Plus computer program.
• To validate the association between computer-mediated closed-ended and open-ended standardized patient assessments.
• To explore the relationship between assessment performance and academic metrics.

Main Methods:

• Development of the SOAP Note Plus computer program for standardized patient assessment.
• A validation study using third-year medical students.
• Analysis of closed-ended and open-ended assessment data using correlation and GENOVA statistics.
• Examination of relationships with undergraduate GPA, medical school GPA, and clinical rotation evaluations.

Main Results:

• The open-ended assessment demonstrated a positive relationship with actual clinical rotation evaluations.
• A positive association was found between the open-ended assessment and undergraduate GPA.
• The open-ended assessment also correlated positively with first two-year medical school GPA.

Conclusions:

• The SOAP Note Plus program facilitates standardized patient assessment.
• Open-ended assessments in this computer-mediated format show promise in reflecting clinical performance.
• Computer-based open-ended assessments may offer a more valid measure of clinical reasoning than closed-ended questions.