Misleading medical literature: An observational study

Language that implies a conclusion not supported by the evidence is common in the medical literature. The hypothesis of the present study was that medical journal publications are more likely to use misleading language for the interpretation of a demonstrated null (i.e. chance or not statistically significant) effect than a demonstrated real (i.e. statistically significant) effect.

This was an observational study of the medical literature with a systematic sampling method. Articles published in The Journal of the American Medical Association, The Lancet and The New England Journal of Medicine over the last two decades were eligible. The language used around the P-value was assessed for misleadingness (i.e. either suggesting an effect existed when a real effect did not exist or vice versa).

There were 228 unique manuscripts examined, containing 400 statements interpreting a P-value proximate to 0.05. The P-value was between 0.036 and 0.050 for 303 (75.8%) statements and between 0.050 and 0.064 for 97 (24.3%) statements. Forty-four (11%) of the statements were misleading. There were 40 (41.2%) false-positive sentences, implying statistical significance when the P-value was >0.05, and four (1.3%) false-negative sentences, implying no statistical significance when the P-value <0.05 (relative risk 31.2; 95% confidence interval 11.5-85.1; P < 0.0001). The proportion of included manuscripts containing at least one misleading sentence was 16.2% (95% confidence interval 12.0-21.6).

Among a random selection of sentences in prestigious journals describing P-values close to 0.05, 1 in 10 are misleading (n = 44, 11%) and this is more prevalent when the P-values are above 0.05 compared to below 0.05. Caution is advised for researchers, clinicians and editors to align with the context and purpose of P-values.