OptimCLM: Optimizing clinical language models for predicting patient outcomes via knowledge distillation, pruning and...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare
• Machine Learning for Clinical Decision Support

Background:

• Clinical Language Models (CLMs) offer potential for healthcare transformation through improved decision-making and resource management.
• High computational costs during inference currently limit the real-world application of CLMs.
• Optimizing CLMs is crucial for enabling widespread adoption in clinical practice.

Purpose of the Study:

• To develop and validate an efficient framework for compressing CLMs.
• To reduce inference time and storage requirements of CLMs without compromising performance.
• To facilitate the deployment of CLMs in real-world clinical environments.

Main Methods:

• The OptimCLM framework utilizes ensemble learning, knowledge distillation (KD), pruning, and quantization.
• Domain-adaptive CLMs (DischargeBERT, COReBERT) were used as a teacher ensemble.
• Knowledge was transferred to smaller models (BERT-PKD, TinyBERT) using black-box KD, pruning, and 8-bit quantization.

Main Results:

• OptimCLM achieved up to 22.88x compression and 28.7x inference speedup.
• Minimal performance degradation observed: <5% AUROC loss for TinyBERT and <2% for BERT-PKD.
• Optimized models outperformed several state-of-the-art models in clinical outcome prediction tasks.

Conclusions:

• Domain-specific fine-tuning combined with ensemble learning and KD is superior to pre-training for knowledge transfer.
• The study demonstrates the feasibility of deploying computationally efficient CLMs in healthcare.
• Optimized CLMs can be developed using reduced computational resources, paving the way for broader clinical integration.