Developing and evaluating a rubric to assess medical students' analytic thinking and predict academic risk

Analytic reasoning requires learners to integrate knowledge, demonstrate analytical skills, and employ metacognition to avoid cognitive biases. Mechanism Maps (MMs), student-generated concept maps, support this process by requiring students to link clinical findings with pathophysiologic mechanisms. However, broader adoption of MMs has been limited by challenges in assessment: existing scoring approaches emphasize structural features rather than accuracy of reasoning, show variable interrater reliability and are time-intensive. Importantly, it remains unclear whether MM-based assessments can identify students at risk of difficulty in analytic reasoning before exam performance.

The authors developed a five-domain rubric (comprehensiveness, analytical detail, organization, accuracy, overall quality) to assess MMs. First-year medical students (n = 134) in a systems-based physiology/pathophysiology course at Harvard Medical School in February-March 2019 created three MMs for clinical vignettes as ungraded exercises. Each MM was assigned one week before a course assessment.

Rater training used an initial set of MMs, after which four trained raters, working in pairs, independently scored the remaining maps. Scoring showed good-to-excellent agreement between raters across most domains, with lowest inter-rater agreement for the accuracy domain. MM scores were associated with later exam performance. For Vignette 2, each one-point increase in score roughly doubled the odds of scoring in the top three exam quartiles rather than the bottom quartile, and this association held even after accounting for quiz performance. Vignette 3 scores showed a similar pattern, though the association was attenuated after adjustment for quiz performance.

The multidimensional rubric enabled reliable MM scoring, associated with exam performance and suggests potential early identification of lower-performing students. Future work should consider prospective validation in additional institutional contexts.