To the Point: Substituting SOAP Notes for Vignettes in Preclinical Assessment Question Stems

Area of Science:

• Medical Education
• Assessment and Evaluation
• Educational Psychology

Background:

• Vignette-based multiple-choice questions (MCQs) are standard in medical assessments.
• Construct-irrelevant variance (CIV) in question stems can inflate difficulty by assessing reading speed over knowledge.
• The Subjective, Objective, Assessment, and Plan (SOAP) note format is a potential method to reduce CIV.

Purpose of the Study:

• To investigate if using SOAP note format for question stems reduces CIV and decreases student answering time.
• To compare the difficulty and discrimination indices of SOAP note versus vignette-based MCQs.

Main Methods:

• Second-year osteopathic medical students were tested using either vignette-based or SOAP note MCQs.
• A multi-level, mixed-effects model analyzed the time difference, controlling for question difficulty, academic rank, and exam timing.
• Difficulty and discrimination indices were compared between question types.

Main Results:

• Students took an average of 3.77 seconds less to answer SOAP note questions compared to vignette-based questions (p < 0.018).
• No significant differences were found in question difficulty or discrimination indices between the two formats.
• Answering time did not vary significantly based on academic rank or time of year.

Conclusions:

• SOAP note question stems reduce construct-irrelevant variance, allowing for more focused assessment of medical knowledge.
• This format enhances assessment efficiency by decreasing the time required for students to answer questions.