To the Point: Substituting SOAP Notes for Vignettes in Preclinical Assessment Question Stems

Vignette-based, multiple-choice questions (MCQs) are common in medical school assessments and licensing exams. Ideally each MCQ assesses a specific objective, but question stems may inadvertently assess reading speed and comprehension by introducing construct-irrelevant variance (CIV). The authors hypothesized that question stems written as a Subjective, Objective, Assessment, and Plan (SOAP) note would address this by reducing CIV resulting in a reduction in time students needed to answer the question.

Vignettes were re-written with the same content presented as a SOAP note. Each of ten exams administered to second-year osteopathic medical students over the academic year possessed four questions in which half the students were tested using a vignette and half the SOAP note version. Data were analyzed using a multi-level, mixed-effects model with question type the main effect; question difficulty, student academic rank, and time during the academic year of exam administration served as covariates. Additional analyses included comparisons of mean difficulty indices and discrimination indices for the two question types.

Students averaged 3.77 s longer on each vignette-based question compared to the SOAP note counterparts (p < 0.018). Average difficulty indices and point biserials did not differ between SOAP note- and vignette-based questions. Time per question did not change significantly over the year, nor did it vary significantly among students in different academic quartiles.

Reduced CIV in the SOAP note stem sharpens the focus of the question on assessing a specific objective, resulting in students spending less time answering the question.