Estimating the deep replicability of scientific findings using human and artificial intelligence

Area of Science:

• Computational science
• Bibliometrics
• Scientific methodology

Background:

• A significant majority of scientific papers fail replication, undermining research integrity and increasing costs.
• Current methods for assessing replicability, such as manual replication tests and prediction markets, are resource-intensive and limited in scalability.
• Failed replications circulate as rapidly as successful ones, posing a challenge to the scientific literature's reliability.

Purpose of the Study:

• To develop and validate an artificial intelligence (AI) model capable of accurately estimating the replicability of scientific papers.
• To assess the generalizability and performance of the AI model across diverse scientific disciplines and methodologies.
• To identify factors influencing the AI model's predictions and explore potential biases.

Main Methods:

• An AI model was trained using ground truth data from studies that had undergone manual replication tests (passed or failed).
• The model's predictive performance was evaluated on extensive out-of-sample datasets, including manually replicated papers from various fields.
• Analysis of model predictions explored potential biases related to research topics, journals, disciplines, and linguistic features (e.g., persuasion, novelty words, n-grams).

Main Results:

• The AI model demonstrated superior performance compared to the base rate of human reviewers and comparable accuracy to prediction markets in estimating replicability.
• Out-of-sample tests showed strong accuracy levels ranging from 0.65 to 0.78 across diverse scientific domains.
• Model accuracy was higher when trained on a paper's textual content versus its statistical data; n-grams (higher-order word combinations) correlated with replication success.

Conclusions:

• The developed AI model offers a promising, scalable, and efficient method for estimating scientific study replicability.
• The model exhibited no detectable bias based on common research characteristics or specific linguistic cues.
• Integrating AI with human expertise can enhance research confidence, provide self-assessment tools, and manage the growing publication volume effectively.