Prompting Pro Tips! Best Practices for Generating Clinical Narrative Summaries

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Informatics

Background:

• Clinical narrative summarization is vital for the Military Health System.
• Evaluating prompt engineering strategies for large language models (LLMs) is crucial for accurate and unbiased clinical summaries.

Purpose of the Study:

• To compare the performance of detailed versus vague prompts in generating clinical narrative summaries.
• To assess the accuracy, bias, and meaningfulness of LLM-generated summaries.

Main Methods:

• Eight anonymized LLMs were tested with eight clinical scenarios.
• Structured and vague prompt variations were used.
• Volunteer clinicians evaluated responses for conformance, adequacy, hallucinations, and bias.

Main Results:

• Both prompt types showed similar instruction conformance (64%).
• Structured prompts yielded more clinically adequate summaries (39% vs. 9%) but increased hallucinations and bias.
• No tested LLM reliably produced clinically usable summaries.

Conclusions:

• Structured prompts enhance LLM clinical summary quality but necessitate mitigation strategies for hallucinations.
• Prompt engineering best practices improve output, but caution is needed to ensure source document fidelity.
• Further tuning is recommended to achieve reliable clinical utility from LLMs.