The over-representation of significant p values in abstracts compared to corresponding full texts: A systematic...

Abstracts are often the only read summaries of research findings, and it is essential that they accurately represent of the contents of the full text of the randomised control trial (RCT). We investigated whether outcomes in surgical trials were selectively reported in abstracts based on their statistical significance.

To compare the proportion of significant p-values reported in abstracts to their corresponding full texts in surgical RCTs.

A Meta-analysis of 350 full text RCTs conducted on humans that compared a surgical intervention to any other intervention. An electronic search of MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials (CENTRAL) was conducted. All outcomes were extracted from the abstract and the full text. Frequency histograms were used to plot the distribution of numerically reported p-values across the statistical significance spectrum. For each RCT, a 2 × 2 table was populated with that trial's outcomes and whether the outcome was statistically significant (p < 0.05). From each 2 × 2 table, an odds ratio (OR) was calculated describing the association between statistical significance, and reporting in the abstract. ORs were pooled in random effects meta-analysis for an overall estimate of the association.

A total of 8258 reported outcomes were included. Outcomes reported in a surgical RCT abstract had three times the odds of being significant when compared to the corresponding full text (OR = 3.0, 95% confidence interval 2.5-3.6, p < 0.001). This finding was consistent and not subject to heterogeneity (I² = 0%). Both histograms demonstrated a large drop in the frequency of reported p values between 0.04 and 0.05, and after the 0.06 thresholds.

Data presented in abstracts is biased to statistically significant outcomes. Clinicians and policy makers should do not rely solely on information presented in abstracts for their decision-making.