Applying machine learning to enhance core outcome set development: automating the data extraction and classification...

Core Outcome Sets (COS) are essential for standardizing outcome reporting in clinical research, yet their development remains resource-intensive and time-consuming. Traditional COS development requires months of expert work for manual outcome extraction and classification from literature. While machine learning (ML) has shown promise in automating systematic reviews, its application to COS development, particularly for outcome identification and classification, remains underexplored. This study evaluates whether ML models can accurately extract and classify verbatim outcomes from clinical studies according to the COMET taxonomy and determines the amount of manually annotated data needed to support reliable model performance.

We developed an ML pipeline using a dataset of 149 full-text studies on lower limb lengthening surgery. The pipeline comprised a Sentence-BERT-based extraction model for identifying verbatim outcomes and a classification model for assigning outcomes to COMET taxonomy domains. We systematically assessed performance using training sets ranging from 5 to 85 articles to establish a practical threshold for reliable model behavior. Model performance was validated using a 28-article hold-out set with standard metrics: precision, recall, and F1-score.

A training size of 20 articles proved sufficient for stable model performance. The extraction model achieved an F1-score of 94% with precision and recall above 90%. The classification model attained a weighted-average F1-score of 86%, with 87% precision and 88% recall. When applied to the full dataset, the system successfully identified 94% of manually extracted outcomes. The distribution of outcome domains identified by ML closely mirrored manual classification with high accuracy.

This study demonstrates the feasibility of applying ML-based outcome extraction and classification within a specific COS development context for lower limb lengthening surgery. By reducing annotation requirements from 149 to just 20 articles while maintaining high accuracy, our approach offers a scalable, reproducible solution that substantially reduces the manual workload in COS development. This pipeline can play a significant role in streamlining evidence synthesis processes, potentially accelerating the generation of outcome lists for consensus-building exercises in COS development.