Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing.

Area of Science:

• Natural Language Processing
• Machine Learning in Healthcare
• Clinical Informatics

Background:

• Clinical natural language processing (NLP) relies heavily on large labeled datasets, which are scarce and costly in the medical domain.
• Weak supervision and in-context learning with large language models (LLMs) show promise but often underperform supervised methods.
• LLM inference is computationally intensive, posing a barrier to widespread clinical adoption.

Purpose of the Study:

• To develop an efficient and effective NLP approach for the clinical domain using LLMs and weak supervision.
• To reduce the dependency on extensive gold-standard clinical data for training high-performance models.
• To investigate a method that achieves dominant performance with minimal domain expertise and labeled data.

Main Methods:

• A prompt-based approach using a fine-tuned LLM (Llama2-13B) to generate weakly labeled data.
• Training a downstream BERT model using the LLM-generated weakly labeled data.
• Further fine-tuning the weakly supervised BERT model on a small amount of gold-standard clinical data.
• Evaluation on three n2c2 datasets to assess performance against established benchmarks.

Main Results:

• The proposed method, using Llama2-13B for weak supervision, consistently outperformed out-of-the-box PubMedBERT by 4.7-47.9% in F1 scores with as few as 10 gold-standard notes.
• With only 50 gold-standard notes, the developed models achieved performance comparable to fully supervised systems.
• The approach demonstrated robust performance across different clinical datasets.

Conclusions:

• Fine-tuning LLMs for weak supervision offers a powerful solution to data scarcity in clinical NLP.
• This method significantly enhances the performance of downstream models like BERT with minimal labeled data.
• The approach provides a computationally efficient and highly effective strategy for clinical text analysis, reducing the need for extensive manual annotation.