Fine-Tuning and Benchmarking Transformer Models for Multiclass Classification of Clinical Research Papers:...

Area of Science:

• Natural Language Processing
• Machine Learning in Healthcare
• Biomedical Informatics

Background:

• The rapid growth of digital health information necessitates efficient text classification.
• Multiclass classification of study types is crucial for evidence synthesis but remains under-explored.
• Transformer models offer promise for enhancing knowledge translation workflows.

Purpose of the Study:

• To fine-tune and evaluate domain-specific transformer models for multiclass classification of clinical literature.
• To categorize papers into original studies, reviews, guidelines, and nonexperimental studies.
• To identify optimal model configurations for accurate literature classification.

Main Methods:

• Fine-tuned seven transformer models on the McMaster PLUS dataset (162,380 papers).
• Utilized a comprehensive grid search (1890 configurations) for hyperparameter optimization (class weight, learning rate, batch size, etc.).
• Assessed models using 10 metrics, including AUROC, F1-score, and MCC, with external validation on the Clinical Hedges dataset.

Main Results:

• Top models achieved macro AUROC ≥0.99, F1-score ≥0.89, and MCC ≥0.88.
• BioBERT-based models demonstrated superior calibration, especially for original studies and reviews.
• Models struggled with nonexperimental and guideline studies, likely due to class imbalance and heterogeneity.

Conclusions:

• Fine-tuned transformer models, especially BioBERT, are effective for multiclass clinical literature classification.
• Hyperparameter optimization is critical for achieving robust model performance.
• Future work should explore methods to address class imbalance for improved classification of all study types.