Algorithmic identification of discrepancies between published ratios and their reported confidence intervals and...

Area of Science:

• Biomedical Informatics
• Scientific Publishing
• Statistical Analysis

Background:

• Human error rates in complex tasks are known to increase.
• The frequency of errors in published scientific literature, particularly in statistical reporting, remains largely unquantified.
• Peer review may not fully mitigate errors in complex calculations.

Purpose of the Study:

• To quantitatively evaluate the frequency and nature of errors in published statistical calculations.
• To investigate the relationship between calculation complexity and error rates in scientific abstracts.
• To identify factors that may mitigate or exacerbate these errors.

Main Methods:

• Extracted statistical ratios (hazard ratio, odds ratio, relative risk), 95% confidence intervals (CIs), and P-values from MEDLINE abstracts.
• Recalculated reported ratios and P-values using extracted CIs.
• Compared error rates between complex ratio-CI-P-value calculations and simpler percent-ratio pairs.

Main Results:

• Analyzed over 486,000 published values for discrepancies.
• Discrepancies were less frequent in simpler percent-ratio calculations (2.7%) compared to ratio-CI-P-value calculations (5.6-7.5%).
• Systematic discrepancies were higher for complex tasks (14.3%) than simple ones (6.7%), and decreased with higher journal impact factors and author counts.

Conclusions:

• Developed a quantitative method to assess published calculation errors.
• Demonstrated that calculation complexity is associated with increased error frequency in scientific literature.
• Highlighted the need for improved methods to detect and reduce statistical reporting errors.