Limitations of Transformers on Clinical Text Classification.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Machine Learning

Background:

• Bidirectional Encoder Representations from Transformers (BERT) models excel at many Natural Language Processing (NLP) tasks.
• However, BERT's fixed input length limits its application to long clinical documents.
• Classifying lengthy clinical texts requires specialized approaches to overcome BERT's limitations.

Purpose of the Study:

• To adapt BERT for document classification on clinical texts exceeding its default input length.
• To compare the performance of scaled BERT methods against simpler baseline architectures.
• To analyze the impact of BERT's pretraining and tokenization on clinical text classification.

Main Methods:

• Developed four methods to scale BERT for processing long clinical texts.
• Compared scaled BERT approaches against a word-level convolutional neural network and a hierarchical self-attention network.
• Evaluated performance on MIMIC-III discharge summaries and SEER cancer pathology reports.

Main Results:

• BERT-based methods did not consistently outperform simpler baseline models on long clinical text classification.
• Simpler architectures, like CNNs and hierarchical attention networks, achieved competitive or superior results.
• Analysis indicated BERT's pretraining and WordPiece tokenization may impede performance on long documents.

Conclusions:

• Standard BERT scaling methods are not always effective for classifying long clinical documents.
• Simpler NLP models can be more suitable for clinical text classification tasks where identifying key phrases is crucial.
• BERT's core components may require modification for optimal performance on extensive clinical text data.