Evidence reversals in primary care research: a study of randomized controlled trials

Evidence-Based Medicine is built on the premise that clinicians can be more confident when their decisions are grounded in high-quality evidence. Furthermore, evidence from studies involving patient-oriented outcomes is preferred when making decisions about tests or treatments. Ideally, the findings of relevant and valid trials should be stable over time, that is, unlikely to be reversed in subsequent research.

To evaluate the stability of evidence from trials relevant to primary healthcare and to identify study characteristics associated with their reversal.

We studied synopses of randomized controlled trials (RCTs) published from 2002 to 2005 as "Daily POEMs" (Patient Oriented Evidence that Matters). The initial evidence (E1) from these POEMs (2002-2005) was compared with the updated evidence (E2) on that same topic in a summary resource (DynaMed 2019). Two physician-raters independently categorized each POEM-RCT as (i) reversed when E1 ≠ E2, or as (ii) not reversed, when E1 = E2. For all "Evidence Reversals" (E1 ≠ E2), we assessed the direction of change in the evidence.

We evaluated 408 POEMs on RCTs. Of those, 35 (9%; 95% confidence interval [6-12]) were identified as reversed, 359 (88%) were identified as not reversed, and 14 (3%) were indeterminate. On average, this represents about 2 evidence reversals per annum for POEMs about RCTs.

Over 12-17 years, 9% of RCTs summarized as POEMs are reversed. Information alerting services that apply strict criteria for relevance and validity of clinical information are likely to identify RCTs whose findings are stable over time.