SPeC: A Soft Prompt-Based Calibration on Performance Variability of Large Language Model in Clinical Notes...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Electronic Health Records (EHRs) are vital for clinical decision-making.
• Summarizing clinical notes enhances patient care by highlighting risks and improving data accessibility.
• Large Language Models (LLMs) show promise in clinical note summarization.

Purpose of the Study:

• To address performance variance in LLM-based clinical note summarization caused by instruction prompt variations.
• To introduce a model-agnostic pipeline, Soft Prompt-Based Calibration (SPeC), to mitigate summarization inconsistencies.

Main Methods:

• Developed a Soft Prompt-Based Calibration (SPeC) pipeline.
• Utilized soft prompts to standardize LLM outputs for summarization tasks.
• Evaluated the method across multiple clinical note datasets and various LLMs.

Main Results:

• The SPeC pipeline significantly reduced performance variance in clinical note summarization.
• SPeC maintained the high efficacy of prompt-based summarization while improving consistency.
• Experimental findings demonstrated robust performance and variance regulation across different LLMs.

Conclusions:

• SPeC offers a reliable and consistent approach to summarizing critical medical information from EHRs.
• This method enhances the practical application of LLMs in healthcare settings.
• Improved summarization accuracy and reliability can lead to reduced medical errors and better patient outcomes.