Paper mill or paper mine? A tentative answer to the sharp increase in research papers based on the Global Burden of...

Recent studies have raised concerns arising from the exploitation ("mining") of public health databases for low-quality, mass-produced papers. However, it remains challenging to disambiguate whether such papers originate from paper mills (commercial entities that sell authorships on mass-produced papers) or the uncoordinated action of individuals facilitated by artificial intelligence (AI) tools and templated workflows. Our study aims to address this question for one particular database, the Global Burden of Disease (GBD) study. We selected this database after noticing that one of our papers on Bayesian age-period-cohort models has recently been experiencing a rapid surge in geographically clustered citations from GBD papers with Chinese affiliations.

We collected bibliometric and article-level metadata from GBD papers to search for indicators of mass-produced research. Moreover, we assessed 713 full-text articles for reported R versions, availability of code and data, and declaration of generative AI use. For 180 articles, we qualitatively screened the figures for graphical similarities. Finally, we conducted an exploratory scoping investigation of online platforms (social media sites, vendor websites) dedicated to do-it-yourself workflows for secondary analyses of public health data.

Although we cannot rule out paper mill involvement, our findings suggest that the geographically clustered increase in GBD publications from China is at least partially driven by independent authors. The wide variety of R versions listed in 477 articles points against centralized paper production. Moreover, despite broad graphical similarities in figure styles that suggest the use of shared visualization tools, substantial variation in ancillary details suggest independent authors finalizing figures. This is corroborated by the identification of an online ecosystem of proprietary tools and services specializing in streamlined do-it-yourself workflows for conducting, writing, and publishing secondary analyses of public health data.

Appropriate efforts should be directed toward evaluating the quality of the identified workflows. Stakeholders in scientific integrity should monitor online platforms dedicated to the rapid production of papers, especially in light of the increasing focus on AI-assisted workflows. Code sharing should be mandated for data-driven secondary analyses. Paywalls and proprietary software licenses hinder transparency and reusability, underscoring the importance of free and open-source software for trustworthy and reproducible research.

There is growing concern that public health databases are being exploited for low-quality, mass-produced scientific papers. However, it remains difficult to distinguish whether these papers originate from organized businesses ("paper mills") or from independent authors using new technologies such as AI for paper writing. When we noticed that one of our papers describing a statistical analysis method suddenly experienced a steep increase in citations, mostly from papers using our method to analyze data from the "Global Burden of Disease" database, we decided to investigate the matter more closely. Our investigations included checking 713 papers for reported software versions and visually comparing the figures in 180 papers. We found that almost no papers featured publicly available analysis code. However, a wide variety of versions of the R programming language were reported in the papers. Moreover, although we found broad graphical similarities in the figures that suggest the use of shared visualization tools, small differences in details indicate that individual authors made final changes. Although we cannot rule out the involvement of organized paper mills, our evidence suggests that the steep increase in papers is at least partially driven by independent authors using a common set of tools. We therefore conducted a follow-up investigation to find out what those tools could be. This led us to an online ecosystem of commercial tools and services designed to help individuals rapidly generate papers based on public health data. Based on our findings, we recommend that the scientific community further investigate these tools and services to ensure high scientific standards are met. We advise against researchers using commercial tools that lack publicly available source code and make it difficult for others to verify the reported findings. We urge journals to require researchers to publicly share their analysis code upon article submission, especially when those articles rely solely on publicly available data.