Development of an assessment technique for basic science retention using the NBME subject exam data

Area of Science:

• Medical Education
• Cognitive Psychology
• Health Professions Education

Background:

• Assessing basic science knowledge retention in medical education presents challenges.
• National Board of Medical Examiners (NBME) clerkship exams reflect preclinical basic science knowledge.
• Effective assessment methods are crucial for tracking knowledge retention during clinical training.

Purpose of the Study:

• To determine if cognitive diagnostic assessment (CDA) can analyze basic science knowledge retention using NBME subject exam data.
• To pilot a CDA model for evaluating student mastery of specific content areas during clerkships.
• To explore the application of CDA for identifying learning gaps and informing curriculum adjustments.

Main Methods:

• Acquired NBME item analysis reports for pediatric clerkship exams (2017-2020).
• Developed a question-by-content Q-matrix linking exam items to necessary skills and content.
• Applied the DINA (deterministic input, noisy "and" gate) cognitive diagnostic model to analyze student content mastery in 12 basic science areas.

Main Results:

• Identified specific strengths and weaknesses in basic science content areas for students in the pediatric clerkship.
• Demonstrated varying levels of student mastery, e.g., "Reproductive systems" (83.8% ± 2.2%) vs. "Skin and subcutaneous tissue" (60.7% ± 3.2%).
• The CDA approach provided quantitative insights into knowledge retention at the content level.

Conclusions:

• The pilot study validates CDA as a technique for quantitatively measuring basic science knowledge retention during clerkships.
• Aggregated data across clerkships can enable comparisons of content areas and identify inter-clerkship variations.
• This method facilitates longitudinal tracking of student performance and guides targeted curricular improvements for enhanced medical education quality.