Machine Learning-Based Identification of Natural History Studies in Rare Diseases: A Step toward Understanding...

Area of Science:

• Medical Informatics
• Computational Biology
• Rare Disease Research

Background:

• Natural history studies (NHS) are vital for rare disease research, aiding in prevalence estimation and biomarker identification.
• Systematic identification of NHS is crucial for large-scale analysis supporting drug development.
• Current methods for identifying NHS are often manual and time-consuming.

Purpose of the Study:

• To develop and evaluate a machine learning-based approach for automated identification of NHS from PubMed.
• To establish a foundational step for large-scale NHS analysis in rare disease drug development.
• To compare the performance of binary versus multiclass classification for NHS identification.

Main Methods:

• A manually curated corpus of NHS was used to train and evaluate machine learning and deep learning models.
• Both binary (NHS-relevant vs. NHS-irrelevant) and multiclass (Unrelated, Irrelevant, Secondary, Primary) classification approaches were tested.
• The PubMedBERT-base-uncased-abstract model was utilized for classification.

Main Results:

• Binary classification models significantly outperformed multiclass models in identifying NHS.
• The PubMedBERT-base-uncased-abstract model achieved the highest performance in binary classification (Precision=0.8171, Recall=0.8079, F1=0.8125, AUCPR=0.8768).
• Deep learning models demonstrate the feasibility of automated NHS identification.

Conclusions:

• Automated NHS identification using deep learning is feasible and effective.
• Binary classification is a highly effective initial strategy for identifying NHS.
• This approach can accelerate data collection for NHS analysis, enhancing understanding of disease progression in rare and common diseases.