Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language...

Area of Science:

• Natural Language Processing
• Machine Learning in Healthcare
• Clinical Informatics

Background:

• Clinical notes contain valuable information hindered by complex language.
• Identifying note sections improves information extraction and clinical NLP tasks.
• Current methods struggle with domain-specific and context-sensitive clinical text.

Purpose of the Study:

• To investigate a dynamic example selection prompting method for section classification.
• To assess the viability of lightweight, open-source large language models (LLMs) for clinical NLP.
• To provide a practical solution for real-world healthcare clinical NLP systems.

Main Methods:

• Developed a dynamic few-shot prompting approach using transformer-based embeddings and a vector store.
• Retrieved similar contextual embeddings for section classification during inference.
• Evaluated the technique on two datasets with varying section schemas and context levels.
• Compared performance against zero-shot and static few-shot baselines.

Main Results:

• Dynamic few-shot prompting achieved the highest F1 scores across all evaluated LLMs and datasets.
• Demonstrated an average macro F1 increase of 39.3% over zero-shot baselines.
• Showcased a 21.1% improvement over static few-shot baselines in the primary task.
• Performance gains were further enhanced by incorporating section context.

Conclusions:

• Dynamically selecting examples for few-shot LLM prompting substantially improves performance.
• Incorporating section context further boosts accuracy in clinical text classification.
• This method holds significant potential for advancing clinical NLP applications.