Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare

Background:

• Large language models (LLMs) excel with public data but struggle with private sources like electronic health records (EHRs).
• Prompt engineering is a potential method to enhance LLM accuracy for EHR data interpretation without domain expertise.

Purpose of the Study:

• To systematically test prompt engineering techniques for improving LLM accuracy in interpreting EHRs for nuanced diagnostic questions.
• To evaluate if prompt engineering can enable LLMs to identify clinical endpoints from EHRs with expert-level accuracy.

Main Methods:

• Designed and tested prompt engineering strategies on GPT-4-turbo using 490 EHR notes from 125 patients with heart rhythm disorders.
• Compared LLM performance against rule-based NLP and BERT-based models for identifying recurrent arrhythmias.
• Validated findings across GPT-3.5-turbo and Jurassic-2 LLMs.

Main Results:

• Out-of-the-box GPT-4-turbo accuracy was 64.3%, increasing to 91.4% with prompt engineering (rationale, structured output, exemplars).
• Prompt-engineered LLM performance significantly surpassed traditional NLP and BERT-based models (P<0.05).
• Consistent accuracy improvements were observed across different LLMs.

Conclusions:

• Prompt engineering enables LLMs to accurately identify clinical endpoints from EHRs, surpassing NLP methods.
• LLM accuracy approximated expert performance without requiring domain-specific knowledge.
• These prompt engineering strategies can be applied to other domains for non-expert automated data analysis.