Using LLMs to Interpret Arterial Blood Gases: Comparison of a Novel Math Scratchpad with Different Prompting Methods...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) show promise in healthcare but struggle with precise calculations, limiting their use in clinical decision support systems (CDSS).
• Accurate interpretation of Arterial Blood Gases (ABGs) requires complex calculations, posing a challenge for current LLM capabilities.
• Existing LLM approaches for medical data analysis often lack the computational precision needed for critical diagnostic tasks.

Purpose of the Study:

• To evaluate a novel method integrating a custom math scratchpad with LLMs for enhanced Arterial Blood Gas (ABG) interpretation.
• To compare the diagnostic accuracy of different LLM prompting strategies, including zero-shot, Retrieval-Augmented Generation (RAG), and a combined approach with a math scratchpad.
• To address the limitations of LLMs in performing domain-specific calculations within a CDSS context.

Main Methods:

• Developed and implemented a novel math scratchpad integrated into an LLM-based CDSS.
• Compared three interpretation methods: zero-shot prompting, prompt engineering with RAG, and a hybrid approach combining the math scratchpad, RAG, and prompt engineering.
• Evaluated the methods on a dataset of 50 Arterial Blood Gas (ABG) results.

Main Results:

• The LLM-integrated CDSS using the novel math scratchpad (Method 3) achieved an accuracy of 86% (43/50), with a 95% confidence interval of 74%-93%.
• Method 2 (RAG and prompt engineering) resulted in 78% accuracy (39/50) [CI 65%-87%].
• Method 1 (zero-shot prompting) demonstrated significantly lower accuracy at 48% (24/50) [CI 35%-61%].

Conclusions:

• The integration of a math scratchpad effectively overcomes the computational limitations of LLMs in ABG interpretation.
• The novel hybrid method significantly enhances the accuracy of LLM-based CDSS for complex clinical data analysis.
• Further validation with real-world patient ABG data is recommended to confirm the clinical utility of this approach.