Should multiple-choice questions get the SAQ? Development of a short-answer question writing rubric

Short-answer questions (SAQs) are often used to assess pharmacy student competency. However, the literature lacks guidance on SAQ development strategies, resulting in varying practices between SAQ writers. Understanding student and faculty perceptions of what constitutes a high-quality SAQ can identify best practices for SAQ development.

We surveyed second-year pharmacy students at the University of California San Francisco (UCSF) to assess their perceptions of SAQs. Likert-type data were descriptively analyzed, and open-ended responses were analyzed using thematic analysis; we used these results to draft an initial SAQ checklist. We then conducted focus groups of UCSF pharmacy faculty to explore their experiences writing SAQs. Transcripts were analyzed using the survey codebook and de novo codes to generate themes. We used the focus group findings to finalize the checklist.

Seventy-five students (82%) completed the survey. Students identified "structure" (organizing into sections/lists) and "content" (clearly delineating student's task) as two ways to improve SAQ quality. Eight faculty participated in focus groups of two to three participants each. Faculty expanded on these previous themes and also identified a new theme, "process." This included peer review of SAQs as well as the iterative process involved in writing the SAQ, model answer, and grading rubric.

Content, structure, and process were the three areas identified for the improvement of SAQ quality at our institution. A checklist outlining best practices in these areas may be best implemented and adopted within the SAQ peer-review process.