Improving model transferability for clinical note section classification models using continued pretraining | JoVE Visualize

Area of Science:

Natural Language Processing (NLP)
Machine Learning in Healthcare
Clinical Informatics

Background:

Accurate classification of clinical note sections (e.g., Subjective, Objective, Assessment, Plan - SOAP) is crucial for downstream NLP tasks like social determinants of health and temporal information extraction.
Clinical note section classification models often exhibit poor transferability across different healthcare institutions, leading to significant accuracy degradation.
Existing methods struggle with cross-domain generalization, even with advanced neural network architectures.

Purpose of the Study:

To develop and evaluate methods for classifying clinical note sections within the SOAP framework that demonstrate improved transferability across institutions.
To investigate the impact of continued pretraining strategies on the cross-domain performance of clinical note section classification models.

Main Methods:

Baseline models were fine-tuned using BERT-based architectures.
Continued pretraining techniques, including domain-adaptive pretraining and task-adaptive pretraining, were employed to enhance model transferability.
The effect of incorporating in-domain annotated samples during fine-tuning was analyzed across varying sample sizes.
The performance improvement from continued pretraining was quantified in terms of the equivalent number of in-domain annotated samples.

Main Results:

Continued pretraining alone did not significantly improve model performance.
When combined with a small number of in-domain annotated samples, continued pretraining substantially improved the F1 score from 0.756 to 0.808 (averaged across 3 datasets).
This performance gain was found to be equivalent to adding approximately 35 in-domain annotated samples.

Conclusions:

Continued pretraining is an effective strategy for enhancing the transferability of clinical note section classification models, particularly in cross-domain scenarios.
The benefits of continued pretraining are most pronounced when augmented with even a limited amount of in-domain labeled data.
This approach offers a viable method to improve the generalizability of NLP models in healthcare settings without requiring extensive in-domain data annotation.