A comparative study of pretrained language models for long clinical text

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Biomedical Informatics

Background:

• Clinical transformer models like ClinicalBERT excel at clinical NLP tasks.
• However, their full self-attention mechanism limits performance on long clinical texts due to high memory consumption.
• Existing models struggle to capture long-term dependencies in extensive clinical notes.

Purpose of the Study:

• To address the limitations of short-sequence clinical NLP models.
• To enhance the modeling of long-term dependencies in long clinical texts.
• To introduce and evaluate novel, domain-enriched, long-sequence transformer models for clinical NLP.

Main Methods:

• Leveraged long-sequence transformer architectures (Longformer, BigBird) with extended input lengths (up to 4096 tokens).
• Developed and pretrained two domain-enriched models: Clinical-Longformer and Clinical-BigBird on a large clinical corpus.
• Evaluated model performance on 10 diverse clinical NLP tasks, including named entity recognition, question answering, and document classification.

Main Results:

• Clinical-Longformer and Clinical-BigBird consistently and significantly outperformed ClinicalBERT and other short-sequence models across all 10 evaluated tasks.
• Achieved new state-of-the-art results in clinical NLP for tasks involving long texts.
• Demonstrated the effectiveness of long-sequence transformers in capturing long-term dependencies within clinical data.

Conclusions:

• Clinical knowledge-enriched long-sequence transformers effectively learn long-term dependencies in long clinical texts.
• The developed models provide a robust foundation for clinical NLP applications utilizing extended text sequences.
• The methodology can inspire the creation of other domain-specific long-sequence transformer models.