Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance...

Area of Science:

• Computational linguistics
• Medical informatics
• Machine learning

Background:

• Clinical natural language processing (NLP) faces challenges in acquiring large datasets for deep learning.
• Billing code data is abundant and can be leveraged for pretraining clinical text encoders.

Purpose of the Study:

• To develop algorithms for encoding clinical text into representations for phenotyping.
• To explore the effectiveness of pretraining clinical text encoders on billing codes.
• To compare universal vs. phenotype-specific encoders for clinical text classification.

Main Methods:

• Pretraining clinical text encoders on large-scale billing code data.
• Exploring various neural encoder architectures.
• Evaluating text representations on clinical text classification tasks, including comorbidity and substance misuse data.

Main Results:

• Successfully trained multiple clinical text encoders.
• Achieved state-of-the-art performance on comorbidity data.
• Demonstrated performance gains on substance misuse data.

Conclusions:

• Pretraining clinical text encoders on billing codes is a promising direction for phenotyping.
• Generated representations exhibit universal properties, benefiting multiple phenotyping tasks.
• Phenotype-specific pretraining can enhance performance for specific tasks at the cost of generality.