Developing healthcare language model embedding spaces.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Artificial Intelligence in Healthcare

Background:

• Large Language Models (LLMs) excel in general NLP but face challenges in specialized domains like healthcare due to data privacy, computational costs, and unique language.
• Traditional fine-tuning of LLMs for healthcare is resource-intensive and often impractical for local healthcare settings.
• Adapting LLMs efficiently to healthcare data is crucial for responsible and sustainable deployment.

Purpose of the Study:

• To develop and evaluate efficient pre-training methods for adapting smaller LLMs to healthcare-specific datasets and tasks.
• To identify pre-training strategies that instill healthcare competency in compact LLMs under computational constraints.
• To enable responsible and sustainable deployment of LLM-based NLP solutions in local healthcare settings.

Main Methods:

• Explored three specialized pre-training methods: Masked Language Modeling (MLM), Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel metadata-based approach.
• Adapted smaller LLMs to various healthcare datasets.
• Assessed performance on downstream document classification tasks using classification accuracy and embedding space analysis.

Main Results:

• Contrastively trained models consistently outperformed other methods in classification tasks, showing strong performance with limited labeled data and fewer updates.
• Domain-adapted LLMs significantly outperformed their general-purpose base models, highlighting the importance of domain specialization.
• Metadata-based pre-training showed interesting embedding cluster separability but did not improve classification accuracy across datasets.

Conclusions:

• Specialized pre-training methods are effective for adapting compact LLMs to healthcare tasks, even with resource constraints.
• Contrastive learning objectives show promise for healthcare NLP, offering efficient adaptation for privacy-sensitive medical tasks.
• This research provides guidelines for pre-training specialized healthcare LLMs, paving the way for more accessible and effective NLP in medicine.