Using LLMs to Interpret Arterial Blood Gases: Comparison of a Novel Math Scratchpad with Different Prompting Methods...

Large language models (LLMs) have demonstrated proficiency in various tasks, yet their effectiveness in a clinical decision support system (CDSS) is evolving. One challenge is their limited ability to perform calculations. This study evaluates a novel method of using a custom math scratchpad to perform domain-specific calculations in interpreting Arterial Blood Gases (ABGs). Three methods are compared: zero-shot prompting (Method 1), prompt engineering with Retrieval-Augmented Generation (RAG) (Method 2), and a combined novel math scratchpad, RAG, and prompt engineering (Method 3). The LLM-integrated CDSS achieved an accuracy rate of 86% (43/50) [confidence interval (CI) 74%-93%] across a dataset of 50 ABG results when utilizing Method 3, compared to 78% (39/50) [CI 65%-87%] for Method 2 and 48% (24/50) [CI 35%-61%] for Method 1. The evaluation demonstrates a math scratchpad's utility in ABG interpretation by overcoming LLMs' calculation limitation. Further testing with real-world patient ABG data is needed.