Evaluating the quality of multiple-choice questions in a NAPLEX preparation book

There is a plethora of preparatory books and guides available to help study for the North American Pharmacist Licensure Examination (NAPLEX). However, the quality of questions included has not been scrutinized. Our objective was to evaluate the quality of multiple-choice questions (MCQs) construction in a commonly used NAPLEX preparatory book.

Five students and two faculty members reviewed MCQs from the RxPrep 2018 edition course book. Item structure and utilization of case-based questions were evaluated using best practices for item construction. Frequency of item writing flaws (IWF) and utilization of cases for case-based questions was identified.

A total of 298 questions were reviewed. Twenty-seven (9.1%) questions met all best practices for item construction. Flawed questions contained an average of 2.53 IWF per MCQ. The most commonly identified best practice violations were answer choices containing differing length and verb tense (21%) and question stems containing too little or too much information necessary to eliminate distractors (16.6%). Of the case-based questions, the majority (61.9%) did not require utilization of the provided case.

This pilot analysis identified that a majority of MCQs in one NAPLEX preparatory source contained IWF. These results align with previous evaluations of test-banks in published books outside of pharmacy. Further evaluation of other preparatory materials, to expand on the findings from this pilot analysis, are needed to evaluate the pervasiveness of IWF in preparatory materials and the effect of flawed questions on utility of study materials.