Text classification models for assessing the completeness of randomized controlled trial publications based on...

Area of Science:

• Medical Informatics
• Clinical Trial Reporting
• Natural Language Processing

Background:

• Complete and transparent reporting of randomized controlled trials (RCTs) is crucial for evaluating study credibility.
• The CONSORT checklist provides essential guidelines for reporting RCTs.
• Automated methods are needed to efficiently assess adherence to CONSORT reporting standards.

Purpose of the Study:

• To develop and compare text classification models for identifying CONSORT checklist items in RCT publications.
• To evaluate the impact of data augmentation techniques on model performance.
• To assess the effectiveness of section-specific models for improved CONSORT item recognition.

Main Methods:

• Trained sentence classification models using PubMedBERT fine-tuning, BioGPT fine-tuning, and GPT-4 in-context learning on a corpus annotated with 37 CONSORT items.
• Applied data augmentation methods including Easy Data Augmentation (EDA), UMLS-EDA, and GPT-4 based text rephrasing.
• Developed and evaluated section-specific PubMedBERT models (e.g., Methods section) and a full model incorporating sentence context and section headers.
• Performed 5-fold cross-validation and reported performance metrics such as precision, recall, F1 score, and AUC.

Main Results:

• The fine-tuned PubMedBERT model, utilizing sentences with surrounding context and section headers, achieved the highest performance (sentence level: 0.71 micro-F1; article level: 0.90 micro-F1).
• Data augmentation methods demonstrated limited positive impact on model performance.
• BioGPT fine-tuning and GPT-4 in-context learning yielded suboptimal results.
• A Methods-specific model improved recognition of methodology-related items, but other section-specific models did not significantly enhance overall performance.

Conclusions:

• A fine-tuned PubMedBERT model effectively recognizes most CONSORT checklist items in RCT publications, though further improvements are possible.
• These advanced models can support journal editorial workflows and CONSORT adherence checks.
• The study highlights the potential of NLP for enhancing the quality and transparency of clinical trial reporting.