Risk of bias assessment in preclinical literature using natural language processing

Area of Science:

• Biomedical Informatics
• Computational Biology
• Research Methodology

Background:

• Systematic reviews of preclinical literature are crucial but time-consuming.
• Accurate risk of bias assessment is essential for reliable evidence synthesis.
• Current methods for risk of bias assessment in preclinical studies can be inefficient.

Purpose of the Study:

• To develop and evaluate natural language processing (NLP) models for automatic risk of bias assessment in preclinical literature.
• To enhance the efficiency and accuracy of systematic reviews.
• To support research improvement and the translation of preclinical findings to clinical applications.

Main Methods:

• Utilized a dataset of 7840 full-text publications from animal experiments with risk of bias annotations.
• Implemented and compared various NLP models: baseline (SVM, logistic regression, random forest), neural networks (CNN, RNN with attention, hierarchical NN), and BERT-based models.
• Tuned hyperparameters and evaluated models using F1 scores on a test set, comparing against a regular expression approach.

Main Results:

• NLP models significantly outperformed regular expressions for four out of five risk of bias items.
• Best model F1 scores: 82.0% (random allocation), 81.6% (blinded outcome assessment), 82.6% (conflict of interests), 91.4% (animal welfare compliance), 46.6% (excluded animals).
• Convolutional neural networks (CNNs) demonstrated overall superior performance, while BERT with sentence extraction excelled for animal welfare regulations.

Conclusions:

• Automated risk of bias assessment using NLP is feasible and effective for preclinical literature.
• The developed models, particularly CNNs, offer a significant improvement over traditional methods.
• The publicly available tool can aid in monitoring research quality and driving improvements in preclinical study reporting.