Evaluating NAPLEX Preparation Exam Questions Generated by a Large Language Model

To assess the quality of North American Pharmacist Licensure Examination (NAPLEX) preparation questions generated by a large language model (LLM).

A commonly available LLM, ChatGPT-4.0, was prompted to develop a 200-question practice NAPLEX based on the 2025 NAPLEX Content Outline. Each question was independently evaluated by 3 college of pharmacy faculty members for correctness and usability, along with other attributes including confirmation of incorrect distractors, reliability, relevance, clarity of language, and whether the question was stated affirmatively. Descriptive statistics were used for data analysis, and Gwet's AC1 statistic was employed to test inter-rater reliability.

Three evaluators found 181 to 191 (91%-96%) of the 200 LLM-generated questions to be correct; 150 to 164 (75%-82%) were either usable as written, and 188 to 190 (94%-95%) were either usable as written or with minor modifications. Gwet's AC1 statistic for the 3 evaluators' ratings for usability was 0.9291 (p <.001), indicating substantial inter-rater reliability. Regarding specific attributes, all correct questions were stated affirmatively, and when averaged across the 3 evaluators, more than 90% were deemed to have incorrect distractors (93%), to be reliable (93%), to be clear (96%), and to be relevant (99%).

Approximately 94% of the NAPLEX-generated questions produced by ChatGPT-4.0 were rated as usable as written or with minor modification by 3 independent evaluators, with a high degree of inter-rater reliability. These findings suggest the potential usefulness of LLMs for generating practice NAPLEX questions for students, though faculty oversight remains important for optimal question development.