Comparison of two validated evidence-based medicine assessments: Do they correlate?

Evidence-based medicine (EBM) has been defined as a process involving five actions: asking, acquiring, appraising, applying, and assessing. Several attempts have been made to create and validate tools to assess EBM aptitude. The newest testing instrument, the ACE tool, which is a 15-question true/false exam, has not been directly compared to the more established Fresno test, which is composed of 12 in-depth short-answer questions. Although both were designed to test Steps 1-4 of the five-step EBM process, it is unclear whether they examine the same things or whether one is superior. To our knowledge there is not a widely used standard for EBM assessment despite the broad requirements for inclusion of EBM in both undergraduate and graduate medical education.

It was hypothesized that these instruments do not correlate between one another, based on inherent differences between them, including assessment format, grading method, and scoring range. The authors sought to examine whether a correlation between the results of these two instruments exists in a population of U.S. medical students.

A retrospective cohort study of 158 fourth-year U.S. medical students in academic year 2018-2019 was conducted. All students were exposed to a focused EBM curriculum, consisting of three guided discussions of separate journal articles clinically relevant to the practice of emergency medicine. Outcomes measured included scores on both the ACE tool and Fresno test using descriptive statistics. Spearman's rho was used to determine the correlation between the ACE and Fresno scores for each student among the entire group. A subgroup analysis was performed to assess for correlations at more extreme data points.

The median scores on the ACE tool and Fresno test were 66.7% and 62.7%. There was no statistically significant correlation between the results of these two assessments (Spearman's rho 0.023, p = 0.774) in our population. The scores from the subgroup of advanced performers on the Fresno test showed a weak statistically significant positive correlation (p = 0.045) to advanced scores on the ACE tool. No other subgroups showed statistically significant correlation.

In our population of U.S. medical students, the results of two known EBM assessment instruments do not correlate with one another. The assessments may differ in what categories of learning they measure or in generalizability or perhaps in what depth of understanding they test overall. Further study is needed to determine what each instrument is measuring and whether there is demonstrable variation across populations.