Automatic categorization of self-acknowledged limitations in randomized controlled trial publications

Area of Science:

• Medical Informatics
• Clinical Trials Research
• Natural Language Processing

Background:

• Acknowledging study limitations is vital for scientific transparency and progress.
• Current reporting of limitations in scientific publications is often insufficient.
• Natural language processing (NLP) offers potential for automated checks to improve research transparency.

Purpose of the Study:

• To develop a dataset and NLP methods for detecting and categorizing self-acknowledged limitations in randomized controlled trial (RCT) publications.
• To improve the accuracy and efficiency of identifying and classifying study limitations.
• To enable large-scale analysis of limitation reporting in clinical research.

Main Methods:

• Created a data model with 15 categories and 24 sub-categories for limitation types.
• Annotated 1090 limitation instances across 200 full-text RCT publications.
• Fine-tuned BERT-based models (PubMedBERT) for sentence and type classification, incorporating data augmentation (EDA, PromDA).
• Applied the best-performing model to approximately 12,000 RCT publications.

Main Results:

• The PubMedBERT model for limitation sentence classification achieved an F1 score of 0.821.
• The best-performing limitation type classification model (PubMedBERT with PromDA) reached an F1 score of 0.7, a 2.7 percentage point improvement.
• Significant improvements in classification accuracy were observed with fine-tuning and data augmentation techniques.

Conclusions:

• Developed NLP models can support automated screening tools for journals to identify reporting issues.
• Automatic extraction of limitations can enhance peer review and evidence synthesis.
• This approach facilitates better searching and aggregation of evidence from clinical trial literature.