Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations

Area of Science:

• Medical research
• Evidence synthesis
• Scientific communication

Background:

• Systematic reviews require timely evidence, but preprint quality varies.
• Preprint articles accelerate knowledge dissemination but lack peer review.
• Assessing preprint quality for systematic reviews is challenging.

Purpose of the Study:

• To develop an automated framework for predicting preprint publication.
• To enhance the efficiency and accuracy of evidence synthesis in systematic reviews.
• To reduce the manual burden of preprint quality assessment.

Main Methods:

• Developed AutoConfidenceScore (automated confidence score assessment) framework.
• Utilized automated data extraction via natural language processing.
• Incorporated semantic embeddings and large language model (LLM)-driven evaluation scores.
• Employed random forest and survival cure models for prediction.

Main Results:

• Random forest classifier achieved an AUROC of 0.747.
• Survival cure model achieved an AUROC of 0.731 for binary outcome.
• Survival cure model achieved a concordance index of 0.667 for time-to-publication risk.

Conclusions:

• AutoConfidenceScore advances preprint prediction through automated extraction and feature integration.
• Combining semantic embeddings and LLM evaluations enhances predictive performance.
• This framework can facilitate preprint incorporation into systematic reviews, improving resource utilization.