Medical physiology faculty overpredict assessment question complexity and alignment with national board expectations

Medical licensing exams require students in allopathic and osteopathic programs to apply foundational science knowledge to board-style patient presentations. This study aimed to assess Medical Physiology course questions and their alignment with the National Board of Medical Examiners' (NBME) Item-Writing Guide. The research team obtained practice and exam questions written by 5 physiology faculty members, totaling 316 practice and 187 exam questions. Questions were reviewed by the research team and assessed for word count and then coded into three categories: basic science (lowest complexity), clinical scenario (moderate complexity), or NBME style (highest complexity). A Step-1 practice exam was obtained from the NBME, and questions were assessed for word count and then coded into the same categories. Faculty members completed a survey that assessed NBME-style question familiarity and estimated the percentages of their practice and exam questions in each category. Faculty predicted their practice and exam questions included 14% and 23%, respectively, of NBME-style questions. However, the research team coded and found that only 3% of the practice and 7% of the exam questions met NBME-style criteria. In contrast, the Step-1 practice exam contained 92% NBME-style questions. Significantly lower word counts were identified in course exams compared to practice Step-1 questions. Survey results indicate a high degree of concern over student success on Step-1 licensure. Faculty predictions highlight extensive overprediction of NBME-style questions and may contribute to gaps in student Step-1 preparedness. Training with the NBME's Item-Writing Guide may be needed to help address this gap.NEW & NOTEWORTHY As national board examinations emphasize patient-centered application, are course exams missing the mark? Analysis of 500+ physiology questions revealed a striking disconnect: faculty believed up to 23% of their exam questions were board style, but fewer than 7% met the standard. Shorter, less complex practice and exam questions may be undermining readiness. These findings expose a hidden gap in assessment design and point to faculty development as a critical step toward improving student success.