Using LLM-generated tools to extract information about reporting statistical software in biomedical and health...

Area of Science:

• Medical Informatics
• Biostatistics
• Scientific Publishing

Background:

• Extracting statistical methodology details from medical literature is challenging and time-consuming.
• Accurate reporting of statistical software is essential for transparency and reproducibility in research.

Purpose of the Study:

• To demonstrate a novel automated procedure for extracting statistical reporting information from published medical articles.
• To identify statistical software packages used in original research articles from the PubMed Central database.

Main Methods:

• Developed a freely available Shiny App utilizing generative artificial intelligence for automated information retrieval.
• Analyzed a large sample of 1740 articles from PubMed Central published in 2021 and 2023.
• Applied the Vibe Coding concept to assess statistical research methods.

Main Results:

• Proprietary statistical software, such as IBM SPSS Statistics, continues to dominate data analysis in published medical research.
• Open-source research software is underutilized despite recommendations for increased adoption.
• A significant number of articles failed to report the statistical software used for analysis.

Conclusions:

• Automated methods can efficiently extract crucial statistical reporting details from large volumes of medical literature.
• There is a persistent reliance on proprietary software and a lack of consistent reporting of analysis tools, hindering reproducibility.
• This study highlights the need for improved adherence to reporting guidelines for statistical software in medical publications.