Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Machine Learning

Background:

• Clinical note section classification is vital for downstream NLP tasks like information extraction.
• Existing models often fail to generalize across different institutions, limiting their real-world applicability.
• The SOAP framework (Subjective, Objective, Assessment, Plan) is a standard for clinical note organization.

Approach:

• Developed and fine-tuned BERT-based models for clinical note section classification.
• Employed continued pretraining strategies, including Domain Adaptive Pretraining (DAPT) and Task Adaptive Pretraining (TAPT), to enhance model transferability.
• Investigated the impact of incorporating out-of-domain annotated samples and varying in-domain sample sizes during fine-tuning.

Key Points:

• Continued pretraining significantly improved model performance (F1 score from 0.756 to 0.808) when supplemented with in-domain annotated samples.
• The performance gain from continued pretraining was equivalent to adding approximately 50 in-domain annotated samples.
• Cross-domain classification remains challenging even with advanced neural network models.

Conclusions:

• Continued pretraining is an effective method for improving the transferability of clinical note section classification models.
• The benefits of continued pretraining are most pronounced when a small quantity of in-domain labeled data is available.
• This research offers a pathway to more robust and generalizable AI tools for clinical text analysis.