Measuring Accuracy of ConsultBot, Hybrid AI Tool, in Interpreting Blood Gas Results

Importance: Arterial and venous blood gases are complex tests used in inpatient settings. While interpretation of blood gas results is algorithmic, it is time consuming and error prone when done manually. Our study measures the accuracy of ConsultBot, in automated interpretation of blood gases. Objective: To determine proportional accuracy of ConsultBot, using rule-based logic combined with a large language model (LLM), in interpreting blood gases. Design: ConsultBot was tested in an IRB-approved single-arm trial using a dataset of 101 blood gas results and compared with a predetermined answer key. The tool's performance was assessed using proportional accuracy, sensitivity, and Cohen's Kappa. Results: ConsultBot achieved a 98% (99/101)[CI 93%-100%] proportional accuracy across the study dataset. Sensitivity was 98%[CI 93%-99%]. Cohen's Kappa of 0.97 suggests high degree of agreement between ConsultBot's interpretations and answer key. Conclusions: ConsultBot's evaluation yielded promising results, demonstrating potential in clinical-decision support for blood gas interpretations.