Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise for analyzing clinical notes.
• Accurate extraction of clinical data is crucial for patient care.
• Automating risk score calculation can streamline clinical workflows.

Purpose of the Study:

• To present a design framework for enhancing LLM accuracy in clinical note analysis.
• To demonstrate the framework's utility via prompt refinement for automated HEART score determination.
• To evaluate the impact of iterative prompt design on LLM performance in extracting complex clinical information.

Main Methods:

• Developed a pipeline for LLM prompt testing using stochastic repeat testing.
• Quantified response errors against physician assessments.
• Iteratively refined prompts for HEART score subcomponents across multiple LLM query rounds.

Main Results:

• Iterative prompt design reduced erroneous non-numerical responses for HEART subscores.
• GPT-4 showed improved accuracy in numerical HEART subscore responses, with mean error decreasing from 0.16 to 0.10.
• The framework facilitated enhanced LLM performance in extracting and applying clinical data.

Conclusions:

• Established a framework for iterative prompt design in clinical applications of LLMs.
• Demonstrated LLMs' potential for structured clinical note analysis.
• Highlighted the need for validation on large-scale, real-world clinical data with privacy safeguards.