A pilot study investigating the relationship between journal impact factor and methodological quality of real-world...

The primary objective of this study was to investigate the association between journal Impact Factor (IF) and study quality in real-world observational studies. The secondary objective was to explore whether the association changes as a function of different study factors (study design, funding type and geographic location).

Study quality was assessed using the Newcastle-Ottawa Scale (NOS). IFs were obtained from journal websites. The association between journal IF and NOS score was evaluated firstly using Spearman's correlation coefficient, and secondly using one-way Analysis of Variance (ANOVA).

We selected 457 studies published in 208 journals across 11 consecutive systematic literature reviews (SLRs) conducted at our organization over the last 5 years. Most studies were cross-sectional and from North America or Europe. Mean (SD) NOS score was 6.6 (1.03) and mean (SD) IF was 5.2 (4.5). Overall, there was a weak positive correlation between NOS score and IF (Spearman's coefficient (ρ) = 0.23 [95% CI: 0.13-0.31]; p < 0.001). There was no correlation between NOS score and IF for prospective cohort studies (ρ = 0.07 [95% CI:-0.12-0.25]) and industry-funded studies (ρ = 0.06 [95% CI:-0.09-0.21]). Based on ANOVA, the effect size, eta squared (η²), was 0.04 (95% CI: 0.01-0.08), indicating a small effect.

While there is some correlation between journal quality and study quality, our findings indicate that high-quality research can be found in journals with lower IF, and assessing study quality requires careful review of study design, methodology, analysis, interpretation, and significance of the findings. Notably, in industry-funded studies, no correlation was found between methodological quality and IF.