Toward assessing clinical trial publications for reporting transparency

Area of Science:

• Biomedical Informatics
• Clinical Trial Methodology
• Natural Language Processing

Background:

• Randomized controlled trials (RCTs) are crucial for evidence-based medicine.
• Ensuring the quality and transparency of RCT reporting is essential for reliable scientific appraisal.
• The CONSORT (Consolidated Standards of Reporting Trials) statement provides guidelines for reporting RCTs.

Purpose of the Study:

• To annotate a corpus of RCT publications with CONSORT checklist items.
• To develop and evaluate text mining methods for automated RCT appraisal using the annotated corpus.
• To create a publicly available resource (CONSORT-TM) for advancing RCT research.

Main Methods:

• Annotation of 50 RCT articles at the sentence level using 37 fine-grained CONSORT checklist items.
• Calculation of inter-annotator agreement using MASI and Krippendorff's α.
• Experimentation with rule-based and supervised learning (SVM, BioBERT) methods for recognizing methodology items.

Main Results:

• Creation of CONSORT-TM, a corpus of 10,709 sentences with 5,246 annotated labels.
• Moderate agreement at article and section levels (MASI: 0.60-0.64), with significant variation among individual items (Krippendorff's α: 0.06-0.96).
• The BioBERT-based model achieved the best performance for methodology items (micro-F1: 0.71), with combined models showing further improvements.

Conclusions:

• CONSORT-TM provides a valuable resource for text mining research on RCT transparency and rigor.
• Low frequency of some CONSORT items poses challenges for model training.
• The corpus can support peer review and authoring assistance, with potential for improved models using larger datasets and scheme modifications.