Paper mills in evidence synthesis: reported involvement among retracted systematic reviews and meta-analyses

To investigate reported paper mill involvement among retracted systematic reviews and meta-analyses (SR/MAs), and to compare retracted SR/MAs with and without reported paper mill involvement with respect to temporal patterns, disciplinary profile, country recorded in the database, and retraction characteristics.

Cross-sectional study based on the Retraction Watch Database. Records were screened to identify retracted SR/MAs. Reported paper mill involvement was assigned only when explicitly stated in the Retraction Watch record or associated retraction information; fake peer review was coded separately. Group comparisons used the Pearson chi-square and Mann-Whitney U tests. Associations with reported paper mill involvement were estimated using univariable Poisson regression with robust variance and reported as prevalence ratios (PRs) with 95% confidence intervals (CIs). These analyses were treated as exploratory, noninferential internal contrasts within the dataset of already retracted SR/MAs.

Among 69,163 retracted publications in the Retraction Watch dataset, 1071 (1.5%) were identified as retracted SR/MAs. Of those, 191 (17.8%) were paper mill-associated and 880 (82.2%) were not. In the database country field, China was recorded for 732/1071 retracted SR/MAs (68.3%) and 177/191 SR/MAs with reported paper mill involvement (92.7%). The annual number of paper mill-associated SR/MAs peaked in 2023 (n = 152). Within this restricted dataset, paper mill-associated SR/MAs had a longer median time to retraction (511 vs 459 days; P = .024), more frequent results-related concerns (89.0% vs 30.0%), data issues (53.4% vs 17.5%), referencing concerns (53.9% vs 13.6%), and ethical concerns (8.9% vs 2.6%) (all P < .001), and less frequent multicountry authorship (8.9% vs 17.6%; P = .004). In univariable Poisson regression analyses restricted to retracted SR/MAs, larger observed proportions of reported paper mill involvement were seen among records linked to China in the database country field (PR: 5.86, 95% CI: 3.45-9.93), more recent publication years (PR 1.20 per 1-year increase, 95% CI 1.16-1.23), the medical broad domain (PR 3.66, 95% CI 1.66-8.08), and records with results-related concerns (PR 11.88, 95% CI 7.68-18.39). These PRs should not be interpreted as likelihoods, risks, or rates beyond the analyzed Retraction Watch subset.

In this Retraction Watch-based analysis, reported paper mill involvement was identified in 191/1071 retracted SR/MAs (17.8%). These records were most often linked to China in the database country field (177/191; 92.7%) and clustered in recent years, particularly 2023. These findings should be interpreted as descriptive patterns within detected and retracted records, not as country-level rates, comparative performance indicators, or estimates of the true prevalence of paper mill activity.

Paper mills are organizations that produce fraudulent scientific papers for payment. Most research on paper mills has focused on primary studies, but less is known about their role in systematic reviews and meta-analyses (SR/MAs), which represent high-level evidence and can directly influence clinical decisions, guidelines, and future research. This study examined retracted SR/MAs in the Retraction Watch Database to describe how often paper mill involvement was reported and how these records differed from other retracted SR/MAs. Among 1071 retracted SR/MAs, 191 were explicitly identified as paper mill-associated. These records were most often linked to China in the database country field, were more common in recent retraction years, and more often involved concerns related to results, data, references, and ethics. These findings describe detected and retracted records indexed in Retraction Watch and should not be interpreted as country-level rates or as the true prevalence of paper mill activity in all SR/MAs. The study highlights the need for clearer retraction notices and better safeguards for identifying unreliable evidence syntheses.